You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 深大李展发 <xl...@foxmail.com> on 2020/05/19 09:57:41 UTC
Cannot access to kafka by server domain and port.
Hi, I have been struggling for this connection problem for a whole week.
I run Kafka on my server machine which is on Cloud. And I cannot manage to connect to Kafka in anyway.
It always print out `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
Here is what I had done to keep it simple so I can find out why:
- I stopped using Docker.
- I configure the Cloud firewall, so it will not block port 9092.
- I start Kafka standalone(1 broker).
- I start zookeeper standalone(1 node).
- Zookeeper and Kafka use JAAS to connect.
- I configure Kafka to log in TRACE level.
- SASL and SSL is all turn off.
I know advertised.listeners is usually the key of these situation, so here is what I had tried:
SITUATION 1:
- set advertised.listener to `localhost` on propose
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client connect to the advertised listener(which is localhost) successfully
4. Topics list is returned, client print out topic list. All well.
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`
1. Kafka even not print out the first metadata request.
2. In client, it print out(Notice, it is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
3. Stop Kafka, start a WWW service on port 9092, can access the WWW service by port 9092.
SITUATION 2:
- set advertised.listener to `xxx.ltd`
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client try to connect to the advertised listener(which is xxx.ltd)
4. Failed, it print out(Notice,it is node 1, not node -1 like above, that means, client is try to connect by the metadata returned by kafka): `Connection to node 1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`:
1. Kafka even not print out the first metadat request.
2. In client, it print out(Notice is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
So, maybe, I think, maybe there is not any TCP request, all request to myServerDomain.ltd/myServerIp:9092 is blocked somehow. So I use `tcpdump -p 9092` to capture packets, here is what I get:
---------------------------------------------
...
2020-05-19 17:34:41.428139 IP 172.18.118.28.9092 > 61.140.182.143.5826: Flags [R.], seq 0, ack 4281665850, win 0, length 0
2020-05-19 17:34:41.842286 IP 61.140.182.143.5828 > 172.18.118.28.9092: Flags [S], seq 3141006320, win 64240, options [mss 1400,sackOK,TS val 1788286298 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:41.842360 IP 172.18.118.28.9092 > 61.140.182.143.5828: Flags [R.], seq 0, ack 3141006321, win 0, length 0
2020-05-19 17:34:42.657551 IP 61.140.182.143.5833 > 172.18.118.28.9092: Flags [S], seq 44626980, win 64240, options [mss 1400,sackOK,TS val 1788287114 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:42.657604 IP 172.18.118.28.9092 > 61.140.182.143.5833: Flags [R.], seq 0, ack 44626981, win 0, length
...
61.140.182.143 is my local laptop. It seems they were communicating.
This is my `server.properties` config:
--------------------------------------------------------
broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka-1
num.partitions=3
num.recovery.threads.per.data.dir=3
offset.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
zookeeper.connect=zookeeper1.myServerDomain.ltd:2181(Original set for Docker network, now I use /etc/hosts map to localhost)
zookeeper.connection.timeout.ms=18000
listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://localhost:9092
advertised.listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://myServerDomain.ltd:9092
inter.broker.listener.name=INSIDE
--------------------------------------------------------
I am almost desperate, ANY SOLUTION OR DIRECTION WOULD BE VERY APPRECIATED!!!
回复： Cannot access to kafka by server domain and port.
Posted by 深大李展发 <xl...@foxmail.com>.
Hi, Moffatt, I had read this article many times, I am horrible and know few in Computer Network, it indeed explains how to configure Kafka to work in AWS and Docker, I will read again. Thank you.
------------------ 原始邮件 ------------------
发件人: "Robin Moffatt"<robin@confluent.io>;
发送时间: 2020年5月19日(星期二) 晚上7:31
收件人: "users"<users@kafka.apache.org>;
主题: Re: Cannot access to kafka by server domain and port.
This should help your understanding:
https://rmoff.net/2018/08/02/kafka-listeners-explained/
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Tue, 19 May 2020 at 11:48, 深大李展发 <xlcyun@foxmail.com> wrote:
> Thank you, Ixy, thanks for your reply, it's working now!
> I will only need to try to set it up in Docker. May I ask why this is
> different? I had also tried "OUTSIDE://:9092" before, is there any
> difference between these config?&nbsp;
> I am always confused by this config, I thought it only means Kafka will
> bind to 9092 port, and accept all connections.&nbsp;
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:&nbsp;"lxy"<lxyscls@163.com&gt;;
> 发送时间:&nbsp;2020年5月19日(星期二) 晚上6:15
> 收件人:&nbsp;"users"<users@kafka.apache.org&gt;;
>
> 主题:&nbsp;Re:Cannot access to kafka by server domain and port.
>
>
>
>
>
>
> It seems that your OUTSIDE listener is wrong. "OUTSIDE://localhost:9092"
> means 127.0.0.1:9092. Try "OUTSIDE://0.0.0.0:9092"
>
>
>
>
>
>
>
>
>
>
> At 2020-05-19 17:57:41, "深大李展发" <xlcyun@foxmail.com&gt; wrote:
>
> Hi, I have been struggling for this connection problem for a whole week.
> I run Kafka&nbsp; on my server machine which is on Cloud. And I cannot
> manage to connect to Kafka in anyway.
> It always print out `Connection to node -1
> (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may
> not be available.`
>
>
>
>
> Here is what I had done to keep it simple so I can find out why:
> - I stopped using Docker.
> - I configure the Cloud firewall, so it will not block port 9092.
> - I start Kafka standalone(1 broker).
> - I start zookeeper standalone(1 node).
> - Zookeeper and Kafka use JAAS to connect.
> - I configure Kafka to log in TRACE level.
> - SASL and SSL is all turn off.
>
>
> I know advertised.listeners is usually the key of these situation, so here
> is what I had tried:
> SITUATION 1:
> - set advertised.listener to `localhost` on propose
> - use `kafka-topic --list --bootstrap-server localhost:9092`:
> &nbsp;&nbsp;&nbsp; 1. Kafka print out the metadata request
> &nbsp;&nbsp;&nbsp; 2. Kafka print out the returned metadata
> &nbsp;&nbsp;&nbsp; 3. Client connect to the advertised listener(which is
> localhost) successfully
> &nbsp;&nbsp;&nbsp; 4. Topics list is returned, client print out topic
> list. All well.
> - use `kafka-topic --list --bootstrap-server xxx.ltd:9092`
> &nbsp;&nbsp;&nbsp; 1. Kafka even not print out the first metadata request.
> &nbsp;&nbsp;&nbsp; 2. In client, it print out(Notice, it is node -1, not
> node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could
> not be established. Broker may not be available.`
> &nbsp;&nbsp;&nbsp; 3. Stop Kafka, start a WWW service on port 9092, can
> access the WWW service by port 9092.
>
>
> SITUATION 2:
> - set advertised.listener to `xxx.ltd`
> - use `kafka-topic --list --bootstrap-server localhost:9092`:
> &nbsp;&nbsp;&nbsp; 1. Kafka print out the metadata request
> &nbsp;&nbsp;&nbsp; 2. Kafka print out the returned metadata
> &nbsp;&nbsp;&nbsp; 3. Client try to connect to the advertised
> listener(which is xxx.ltd)
> &nbsp;&nbsp;&nbsp; 4. Failed, it print out(Notice,it is node 1, not node
> -1 like above, that means, client is try to connect by the metadata
> returned by kafka): `Connection to node 1
> (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may
> not be available.`
> - use `kafka-topic --list --bootstrap-server xxx.ltd:9092`:
> &nbsp;&nbsp;&nbsp; 1. Kafka even not print out the first metadat request.
> &nbsp;&nbsp;&nbsp; 2. In client, it print out(Notice is node -1, not node
> 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not
> be established. Broker may not be available.`
>
>
> So, maybe, I think, maybe there is not any TCP request, all request to
> myServerDomain.ltd/myServerIp:9092 is blocked somehow. So I use `tcpdump -p
> 9092` to capture packets, here is what I get:
> ---------------------------------------------
> ...
> 2020-05-19 17:34:41.428139 IP 172.18.118.28.9092 &gt; 61.140.182.143.5826:
> Flags [R.], seq 0, ack 4281665850, win 0, length 0
> 2020-05-19 17:34:41.842286 IP 61.140.182.143.5828 &gt; 172.18.118.28.9092:
> Flags [S], seq 3141006320, win 64240, options [mss 1400,sackOK,TS val
> 1788286298 ecr 0,nop,wscale 1], length 0
> 2020-05-19 17:34:41.842360 IP 172.18.118.28.9092 &gt; 61.140.182.143.5828:
> Flags [R.], seq 0, ack 3141006321, win 0, length 0
> 2020-05-19 17:34:42.657551 IP 61.140.182.143.5833 &gt; 172.18.118.28.9092:
> Flags [S], seq 44626980, win 64240, options [mss 1400,sackOK,TS val
> 1788287114 ecr 0,nop,wscale 1], length 0
> 2020-05-19 17:34:42.657604 IP 172.18.118.28.9092 &gt; 61.140.182.143.5833:
> Flags [R.], seq 0, ack 44626981, win 0, length
> ...
> 61.140.182.143 is my local laptop. It seems they were communicating.
>
>
> This is my `server.properties` config:
> --------------------------------------------------------
> broker.id=1
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/var/log/kafka-1
> num.partitions=3
> num.recovery.threads.per.data.dir=3
> offset.topic.replication.factor=3
> transaction.state.log.replication.factor=3
> transaction.state.log.min.isr=3
>
>
> zookeeper.connect=zookeeper1.myServerDomain.ltd:2181(Original set for
> Docker network, now I use /etc/hosts map to localhost)
> zookeeper.connection.timeout.ms=18000
>
>
> listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
> listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://localhost:9092
>
> advertised.listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://myServerDomain.ltd:9092
> inter.broker.listener.name=INSIDE
> --------------------------------------------------------
>
>
> I am almost desperate, ANY SOLUTION OR DIRECTION WOULD BE VERY
> APPRECIATED!!!
Re: Cannot access to kafka by server domain and port.
Posted by Robin Moffatt <ro...@confluent.io>.
This should help your understanding:
https://rmoff.net/2018/08/02/kafka-listeners-explained/
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Tue, 19 May 2020 at 11:48, 深大李展发 <xl...@foxmail.com> wrote:
> Thank you, Ixy, thanks for your reply, it's working now!
> I will only need to try to set it up in Docker. May I ask why this is
> different? I had also tried "OUTSIDE://:9092" before, is there any
> difference between these config?
> I am always confused by this config, I thought it only means Kafka will
> bind to 9092 port, and accept all connections.
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "lxy"<lxyscls@163.com>;
> 发送时间: 2020年5月19日(星期二) 晚上6:15
> 收件人: "users"<users@kafka.apache.org>;
>
> 主题: Re:Cannot access to kafka by server domain and port.
>
>
>
>
>
>
> It seems that your OUTSIDE listener is wrong. "OUTSIDE://localhost:9092"
> means 127.0.0.1:9092. Try "OUTSIDE://0.0.0.0:9092"
>
>
>
>
>
>
>
>
>
>
> At 2020-05-19 17:57:41, "深大李展发" <xlcyun@foxmail.com> wrote:
>
> Hi, I have been struggling for this connection problem for a whole week.
> I run Kafka on my server machine which is on Cloud. And I cannot
> manage to connect to Kafka in anyway.
> It always print out `Connection to node -1
> (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may
> not be available.`
>
>
>
>
> Here is what I had done to keep it simple so I can find out why:
> - I stopped using Docker.
> - I configure the Cloud firewall, so it will not block port 9092.
> - I start Kafka standalone(1 broker).
> - I start zookeeper standalone(1 node).
> - Zookeeper and Kafka use JAAS to connect.
> - I configure Kafka to log in TRACE level.
> - SASL and SSL is all turn off.
>
>
> I know advertised.listeners is usually the key of these situation, so here
> is what I had tried:
> SITUATION 1:
> - set advertised.listener to `localhost` on propose
> - use `kafka-topic --list --bootstrap-server localhost:9092`:
> 1. Kafka print out the metadata request
> 2. Kafka print out the returned metadata
> 3. Client connect to the advertised listener(which is
> localhost) successfully
> 4. Topics list is returned, client print out topic
> list. All well.
> - use `kafka-topic --list --bootstrap-server xxx.ltd:9092`
> 1. Kafka even not print out the first metadata request.
> 2. In client, it print out(Notice, it is node -1, not
> node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could
> not be established. Broker may not be available.`
> 3. Stop Kafka, start a WWW service on port 9092, can
> access the WWW service by port 9092.
>
>
> SITUATION 2:
> - set advertised.listener to `xxx.ltd`
> - use `kafka-topic --list --bootstrap-server localhost:9092`:
> 1. Kafka print out the metadata request
> 2. Kafka print out the returned metadata
> 3. Client try to connect to the advertised
> listener(which is xxx.ltd)
> 4. Failed, it print out(Notice,it is node 1, not node
> -1 like above, that means, client is try to connect by the metadata
> returned by kafka): `Connection to node 1
> (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may
> not be available.`
> - use `kafka-topic --list --bootstrap-server xxx.ltd:9092`:
> 1. Kafka even not print out the first metadat request.
> 2. In client, it print out(Notice is node -1, not node
> 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not
> be established. Broker may not be available.`
>
>
> So, maybe, I think, maybe there is not any TCP request, all request to
> myServerDomain.ltd/myServerIp:9092 is blocked somehow. So I use `tcpdump -p
> 9092` to capture packets, here is what I get:
> ---------------------------------------------
> ...
> 2020-05-19 17:34:41.428139 IP 172.18.118.28.9092 > 61.140.182.143.5826:
> Flags [R.], seq 0, ack 4281665850, win 0, length 0
> 2020-05-19 17:34:41.842286 IP 61.140.182.143.5828 > 172.18.118.28.9092:
> Flags [S], seq 3141006320, win 64240, options [mss 1400,sackOK,TS val
> 1788286298 ecr 0,nop,wscale 1], length 0
> 2020-05-19 17:34:41.842360 IP 172.18.118.28.9092 > 61.140.182.143.5828:
> Flags [R.], seq 0, ack 3141006321, win 0, length 0
> 2020-05-19 17:34:42.657551 IP 61.140.182.143.5833 > 172.18.118.28.9092:
> Flags [S], seq 44626980, win 64240, options [mss 1400,sackOK,TS val
> 1788287114 ecr 0,nop,wscale 1], length 0
> 2020-05-19 17:34:42.657604 IP 172.18.118.28.9092 > 61.140.182.143.5833:
> Flags [R.], seq 0, ack 44626981, win 0, length
> ...
> 61.140.182.143 is my local laptop. It seems they were communicating.
>
>
> This is my `server.properties` config:
> --------------------------------------------------------
> broker.id=1
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/var/log/kafka-1
> num.partitions=3
> num.recovery.threads.per.data.dir=3
> offset.topic.replication.factor=3
> transaction.state.log.replication.factor=3
> transaction.state.log.min.isr=3
>
>
> zookeeper.connect=zookeeper1.myServerDomain.ltd:2181(Original set for
> Docker network, now I use /etc/hosts map to localhost)
> zookeeper.connection.timeout.ms=18000
>
>
> listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
> listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://localhost:9092
>
> advertised.listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://myServerDomain.ltd:9092
> inter.broker.listener.name=INSIDE
> --------------------------------------------------------
>
>
> I am almost desperate, ANY SOLUTION OR DIRECTION WOULD BE VERY
> APPRECIATED!!!
回复：Cannot access to kafka by server domain and port.
Posted by 深大李展发 <xl...@foxmail.com>.
Thank you, Ixy, thanks for your reply, it's working now!
I will only need to try to set it up in Docker. May I ask why this is different? I had also tried "OUTSIDE://:9092" before, is there any difference between these config?
I am always confused by this config, I thought it only means Kafka will bind to 9092 port, and accept all connections.
------------------ 原始邮件 ------------------
发件人: "lxy"<lxyscls@163.com>;
发送时间: 2020年5月19日(星期二) 晚上6:15
收件人: "users"<users@kafka.apache.org>;
主题: Re:Cannot access to kafka by server domain and port.
It seems that your OUTSIDE listener is wrong. "OUTSIDE://localhost:9092" means 127.0.0.1:9092. Try "OUTSIDE://0.0.0.0:9092"
At 2020-05-19 17:57:41, "深大李展发" <xlcyun@foxmail.com> wrote:
Hi, I have been struggling for this connection problem for a whole week.
I run Kafka on my server machine which is on Cloud. And I cannot manage to connect to Kafka in anyway.
It always print out `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
Here is what I had done to keep it simple so I can find out why:
- I stopped using Docker.
- I configure the Cloud firewall, so it will not block port 9092.
- I start Kafka standalone(1 broker).
- I start zookeeper standalone(1 node).
- Zookeeper and Kafka use JAAS to connect.
- I configure Kafka to log in TRACE level.
- SASL and SSL is all turn off.
I know advertised.listeners is usually the key of these situation, so here is what I had tried:
SITUATION 1:
- set advertised.listener to `localhost` on propose
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client connect to the advertised listener(which is localhost) successfully
4. Topics list is returned, client print out topic list. All well.
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`
1. Kafka even not print out the first metadata request.
2. In client, it print out(Notice, it is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
3. Stop Kafka, start a WWW service on port 9092, can access the WWW service by port 9092.
SITUATION 2:
- set advertised.listener to `xxx.ltd`
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client try to connect to the advertised listener(which is xxx.ltd)
4. Failed, it print out(Notice,it is node 1, not node -1 like above, that means, client is try to connect by the metadata returned by kafka): `Connection to node 1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`:
1. Kafka even not print out the first metadat request.
2. In client, it print out(Notice is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
So, maybe, I think, maybe there is not any TCP request, all request to myServerDomain.ltd/myServerIp:9092 is blocked somehow. So I use `tcpdump -p 9092` to capture packets, here is what I get:
---------------------------------------------
...
2020-05-19 17:34:41.428139 IP 172.18.118.28.9092 > 61.140.182.143.5826: Flags [R.], seq 0, ack 4281665850, win 0, length 0
2020-05-19 17:34:41.842286 IP 61.140.182.143.5828 > 172.18.118.28.9092: Flags [S], seq 3141006320, win 64240, options [mss 1400,sackOK,TS val 1788286298 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:41.842360 IP 172.18.118.28.9092 > 61.140.182.143.5828: Flags [R.], seq 0, ack 3141006321, win 0, length 0
2020-05-19 17:34:42.657551 IP 61.140.182.143.5833 > 172.18.118.28.9092: Flags [S], seq 44626980, win 64240, options [mss 1400,sackOK,TS val 1788287114 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:42.657604 IP 172.18.118.28.9092 > 61.140.182.143.5833: Flags [R.], seq 0, ack 44626981, win 0, length
...
61.140.182.143 is my local laptop. It seems they were communicating.
This is my `server.properties` config:
--------------------------------------------------------
broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka-1
num.partitions=3
num.recovery.threads.per.data.dir=3
offset.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
zookeeper.connect=zookeeper1.myServerDomain.ltd:2181(Original set for Docker network, now I use /etc/hosts map to localhost)
zookeeper.connection.timeout.ms=18000
listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://localhost:9092
advertised.listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://myServerDomain.ltd:9092
inter.broker.listener.name=INSIDE
--------------------------------------------------------
I am almost desperate, ANY SOLUTION OR DIRECTION WOULD BE VERY APPRECIATED!!!
Re:Cannot access to kafka by server domain and port.
Posted by lxy <lx...@163.com>.
It seems that your OUTSIDE listener is wrong. "OUTSIDE://localhost:9092" means 127.0.0.1:9092. Try "OUTSIDE://0.0.0.0:9092"
At 2020-05-19 17:57:41, "深大李展发" <xl...@foxmail.com> wrote:
Hi, I have been struggling for this connection problem for a whole week.
I run Kafka on my server machine which is on Cloud. And I cannot manage to connect to Kafka in anyway.
It always print out `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
Here is what I had done to keep it simple so I can find out why:
- I stopped using Docker.
- I configure the Cloud firewall, so it will not block port 9092.
- I start Kafka standalone(1 broker).
- I start zookeeper standalone(1 node).
- Zookeeper and Kafka use JAAS to connect.
- I configure Kafka to log in TRACE level.
- SASL and SSL is all turn off.
I know advertised.listeners is usually the key of these situation, so here is what I had tried:
SITUATION 1:
- set advertised.listener to `localhost` on propose
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client connect to the advertised listener(which is localhost) successfully
4. Topics list is returned, client print out topic list. All well.
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`
1. Kafka even not print out the first metadata request.
2. In client, it print out(Notice, it is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
3. Stop Kafka, start a WWW service on port 9092, can access the WWW service by port 9092.
SITUATION 2:
- set advertised.listener to `xxx.ltd`
- use `kafka-topic --list --bootstrap-server localhost:9092`:
1. Kafka print out the metadata request
2. Kafka print out the returned metadata
3. Client try to connect to the advertised listener(which is xxx.ltd)
4. Failed, it print out(Notice,it is node 1, not node -1 like above, that means, client is try to connect by the metadata returned by kafka): `Connection to node 1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
- use `kafka-topic --list --bootstrap-server xxx.ltd:9092`:
1. Kafka even not print out the first metadat request.
2. In client, it print out(Notice is node -1, not node 1): `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not be available.`
So, maybe, I think, maybe there is not any TCP request, all request to myServerDomain.ltd/myServerIp:9092 is blocked somehow. So I use `tcpdump -p 9092` to capture packets, here is what I get:
---------------------------------------------
...
2020-05-19 17:34:41.428139 IP 172.18.118.28.9092 > 61.140.182.143.5826: Flags [R.], seq 0, ack 4281665850, win 0, length 0
2020-05-19 17:34:41.842286 IP 61.140.182.143.5828 > 172.18.118.28.9092: Flags [S], seq 3141006320, win 64240, options [mss 1400,sackOK,TS val 1788286298 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:41.842360 IP 172.18.118.28.9092 > 61.140.182.143.5828: Flags [R.], seq 0, ack 3141006321, win 0, length 0
2020-05-19 17:34:42.657551 IP 61.140.182.143.5833 > 172.18.118.28.9092: Flags [S], seq 44626980, win 64240, options [mss 1400,sackOK,TS val 1788287114 ecr 0,nop,wscale 1], length 0
2020-05-19 17:34:42.657604 IP 172.18.118.28.9092 > 61.140.182.143.5833: Flags [R.], seq 0, ack 44626981, win 0, length
...
61.140.182.143 is my local laptop. It seems they were communicating.
This is my `server.properties` config:
--------------------------------------------------------
broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/log/kafka-1
num.partitions=3
num.recovery.threads.per.data.dir=3
offset.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
zookeeper.connect=zookeeper1.myServerDomain.ltd:2181(Original set for Docker network, now I use /etc/hosts map to localhost)
zookeeper.connection.timeout.ms=18000
listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://localhost:9092
advertised.listeners=INSIDE://kafka1.myServerDomain.ltd:19092,OUTSIDE://myServerDomain.ltd:9092
inter.broker.listener.name=INSIDE
--------------------------------------------------------
I am almost desperate, ANY SOLUTION OR DIRECTION WOULD BE VERY APPRECIATED!!!