You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Wang, Shuo" <wk...@gmail.com> on 2019/10/17 19:13:26 UTC
kafka connection from docker
Hi,
I have a question regarding connecting to kafka broker from docker.
I have zookeeper and kafka broker running on my local machine.
I have a docker container running on the same local machine with
--network=host
I want to send message from inside the docker container to my local kafka
broker.
From inside the docker, I can connect to zookeeper and seek the existing
topicsby running:
`./bin/kafka-topics.sh --zookeeper localhost:2181 --list`
But I cannot connect to the kafka broker with either the
`kafka-console-consumer.sh` or `kafka-console-producer.sh`,
by running : `bin/kafka-console-consumer.sh --bootstrap-server
localhost:9092 --topic test`
I get:
```
[2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1,
groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0
rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: aictjt
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at
org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
at
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
at
kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
```
any help?
Thanks.
-Shuo
Re: kafka connection from docker
Posted by "M. Manna" <ma...@gmail.com>.
Please check your advertised.listeners and listeners config.
Thanks,
On Thu, 17 Oct 2019 at 22:13, Wang, Shuo <wk...@gmail.com> wrote:
> Hi,
>
> I have a question regarding connecting to kafka broker from docker.
>
> I have zookeeper and kafka broker running on my local machine.
> I have a docker container running on the same local machine with
> --network=host
> I want to send message from inside the docker container to my local kafka
> broker.
>
> From inside the docker, I can connect to zookeeper and seek the existing
> topicsby running:
> `./bin/kafka-topics.sh --zookeeper localhost:2181 --list`
>
> But I cannot connect to the kafka broker with either the
> `kafka-console-consumer.sh` or `kafka-console-producer.sh`,
>
> by running : `bin/kafka-console-consumer.sh --bootstrap-server
> localhost:9092 --topic test`
> I get:
>
> ```
> [2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1,
> groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0
> rack: null) (org.apache.kafka.clients.NetworkClient)
> java.net.UnknownHostException: aictjt
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
> at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
> at
>
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
> at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
> at
>
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
> at
>
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
> at
>
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
> at
>
> kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> ```
>
>
> any help?
>
> Thanks.
>
>
>
> -Shuo
>
Re: kafka connection from docker
Posted by Hans Jespersen <ha...@confluent.io>.
This is a great blog post that explains how kafka works with advertised listeners and docker
https://rmoff.net/2018/08/02/kafka-listeners-explained/
-hans
> On Oct 18, 2019, at 5:36 AM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> I do not understand this.
>
> You have on a physical host running zookeeper locally and a broker running
> as well. These are using default physical host portslike 2181, 9092 etc.
>
> Then you have installed zookeeper and kaka broker on dockers?
>
> docker run --net=host -d --name zookeeper -p 2181:2181 -p 2888:2888 -p
> 3888:3888 jplock/zookeeper
>
> docker run -d --net=host --name kafka_broker1 -p 9092:9092 -e
> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
> KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 -e KAFKA_ADVERTISED_PORT=9092
> ches/kafka
>
> Ok --net=host means that the docker will use the host port as its own.
> Indeed other mappings will be discarded. You can either map dockers to
> host port or use other posts but not both!
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>> On Thu, 17 Oct 2019 at 22:13, Wang, Shuo <wk...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have a question regarding connecting to kafka broker from docker.
>>
>> I have zookeeper and kafka broker running on my local machine.
>> I have a docker container running on the same local machine with
>> --network=host
>> I want to send message from inside the docker container to my local kafka
>> broker.
>>
>> From inside the docker, I can connect to zookeeper and seek the existing
>> topicsby running:
>> `./bin/kafka-topics.sh --zookeeper localhost:2181 --list`
>>
>> But I cannot connect to the kafka broker with either the
>> `kafka-console-consumer.sh` or `kafka-console-producer.sh`,
>>
>> by running : `bin/kafka-console-consumer.sh --bootstrap-server
>> localhost:9092 --topic test`
>> I get:
>>
>> ```
>> [2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1,
>> groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0
>> rack: null) (org.apache.kafka.clients.NetworkClient)
>> java.net.UnknownHostException: aictjt
>> at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
>> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
>> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
>> at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
>> at
>>
>> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
>> at
>>
>> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
>> at
>>
>> org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
>> at
>>
>> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
>> at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
>> at
>>
>> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
>> at
>>
>> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
>> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
>> at
>>
>> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
>> at
>>
>> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
>> at
>>
>> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
>> at
>>
>> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
>> at
>>
>> kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
>> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
>> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
>> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
>> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>>
>> ```
>>
>>
>> any help?
>>
>> Thanks.
>>
>>
>>
>> -Shuo
>>
Re: kafka connection from docker
Posted by Mich Talebzadeh <mi...@gmail.com>.
I do not understand this.
You have on a physical host running zookeeper locally and a broker running
as well. These are using default physical host portslike 2181, 9092 etc.
Then you have installed zookeeper and kaka broker on dockers?
docker run --net=host -d --name zookeeper -p 2181:2181 -p 2888:2888 -p
3888:3888 jplock/zookeeper
docker run -d --net=host --name kafka_broker1 -p 9092:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 -e KAFKA_ADVERTISED_PORT=9092
ches/kafka
Ok --net=host means that the docker will use the host port as its own.
Indeed other mappings will be discarded. You can either map dockers to
host port or use other posts but not both!
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 17 Oct 2019 at 22:13, Wang, Shuo <wk...@gmail.com> wrote:
> Hi,
>
> I have a question regarding connecting to kafka broker from docker.
>
> I have zookeeper and kafka broker running on my local machine.
> I have a docker container running on the same local machine with
> --network=host
> I want to send message from inside the docker container to my local kafka
> broker.
>
> From inside the docker, I can connect to zookeeper and seek the existing
> topicsby running:
> `./bin/kafka-topics.sh --zookeeper localhost:2181 --list`
>
> But I cannot connect to the kafka broker with either the
> `kafka-console-consumer.sh` or `kafka-console-producer.sh`,
>
> by running : `bin/kafka-console-consumer.sh --bootstrap-server
> localhost:9092 --topic test`
> I get:
>
> ```
> [2019-10-17 19:12:04,097] WARN [Consumer clientId=consumer-1,
> groupId=console-consumer-99825] Error connecting to node aictjt:9092 (id: 0
> rack: null) (org.apache.kafka.clients.NetworkClient)
> java.net.UnknownHostException: aictjt
> at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
> at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
> at
>
> org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
> at
>
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943)
> at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68)
> at
>
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
> at
>
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
> at
>
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:259)
> at
>
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:326)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
> at
>
> kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:439)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:105)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> ```
>
>
> any help?
>
> Thanks.
>
>
>
> -Shuo
>