You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chuck Musser <cm...@sonic.net> on 2017/04/27 18:08:10 UTC
Controller connection failures
I'm running into a problem with a 3 broker cluster where,
intermittently, one of the broker's controller begins to report that
it cannot connect to the other brokers and repeatedly logs the
failure.
Each broker is running in its own Docker container on separate
machines. These Docker containers have exposed 9092, which I think is
sufficient for operation, but not sure.
The log message are these:
[2017-04-27 17:16:28,985] WARN [Controller-3-to-broker-2-send-thread], Controller 3's connection to broker 64174aa85d04:9092 (id: 2 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 64174aa85d04:9092 (id: 2 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-04-27 17:16:28,986] WARN [Controller-3-to-broker-1-send-thread], Controller 3's connection to broker d4b8943ad4b5:9092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to d4b8943ad4b5:9092 (id: 1 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
This is Kafka 2.12-0.10.2.0. I'm wondering:
1. How do we figure out the cause of the connect failures?
2. What's the controller anyway?
3. Are there some command-line diagnostic tools for inspecting the health of the system?
Thanks for any help,
Chuck
Re: Controller connection failures
Posted by Chuck Musser <cm...@sonic.net>.
Running inside a Docker container introduced a slight wrinkle. It’s easily resolvable. This line from the controller log file was the tip off:
java.io.IOException: Connection to 64174aa85d04:9092 (id: 2 rack: null) failed
64174aa85d04 was a container ID of one of the other brokers. All the brokers were advertising the container ID as their hostname and the controller was not able to resolve any of those names. The advertised hostname:port is set via the "advertised.listeners” configuration variable in server.properties and defaults to the hostname. But inside a Docker container, the hostname defaults to the container ID. This name is not resolvable anywhere but inside the container.
So, If you’re running Kafka in a container and expect other systems to connect to it via the IP address of the Docker host (the host running your container), there are a couple ways to achieve this:
Technique #1 (First solution found, works fine)
1. Leave the “listeners” parameter unconfigured, so that Kafka can bind to the private network address that Docker assigns the container. Docker will take care of routing packets from outside systems to the container’s Kafka process.
2. Set “advertised.listeners" to the hostname of the Docker host, so that the broker can advertise a resolvable name for other systems to contact.
Technique #2 (Discovered a little later, preferable due to less configuration)
1. Simply leave both “listeners” and “advertised.listeners" parameter unconfigured, but run the Docker container with the "—hostname” parameter, specifying the DNS name of the Docker host on which the container is running. This works because “advertised.listeners" will default to the specified name (which is resolvable). Also, Docker will add an entry to “/etc/hosts” that maps this name to the private network address of the container’s interface. So when “listeners” defaults to the same hostname, that name resolves locally to an address in the container and Kafka can successfully bind to this address on startup.
Also, thanks for the link to the controller internals doc. I wanted some more general info on how the system works, so that was informative.
Chuck
> On Apr 30, 2017, at 7:40 AM, Michal Borowiecki <mi...@openbet.com> wrote:
>
> Hi Chuck,
>
> Are you running zookeepers in the same containers as Kafka brokers?
>
> Kafka brokers should be able to communicate with any of the zookeepers and, more importantly, zookeepers need to be able to talk to each-other.
>
> Therefore, the zookeeper port should be exposed too (2181 by default), otherwise you're not going to have a valid cluster.
>
> Docs on the controller are here:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals>
>
> Let me know if that helped.
>
>>
Re: Controller connection failures
Posted by Michal Borowiecki <mi...@openbet.com>.
Hi Chuck,
Are you running zookeepers in the same containers as Kafka brokers?
Kafka brokers should be able to communicate with any of the zookeepers
and, more importantly, zookeepers need to be able to talk to each-other.
Therefore, the zookeeper port should be exposed too (2181 by default),
otherwise you're not going to have a valid cluster.
Docs on the controller are here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals
Let me know if that helped.
Cheers,
Micha\u0142
On 27/04/17 19:08, Chuck Musser wrote:
> I'm running into a problem with a 3 broker cluster where,
> intermittently, one of the broker's controller begins to report that
> it cannot connect to the other brokers and repeatedly logs the
> failure.
>
> Each broker is running in its own Docker container on separate
> machines. These Docker containers have exposed 9092, which I think is
> sufficient for operation, but not sure.
>
> The log message are these:
> [2017-04-27 17:16:28,985] WARN [Controller-3-to-broker-2-send-thread], Controller 3's connection to broker 64174aa85d04:9092 (id: 2 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
> java.io.IOException: Connection to 64174aa85d04:9092 (id: 2 rack: null) failed
> at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
> at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
> at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
> at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
> at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2017-04-27 17:16:28,986] WARN [Controller-3-to-broker-1-send-thread], Controller 3's connection to broker d4b8943ad4b5:9092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
> java.io.IOException: Connection to d4b8943ad4b5:9092 (id: 1 rack: null) failed
> at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
> at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
> at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
> at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
> at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
>
> This is Kafka 2.12-0.10.2.0. I'm wondering:
>
> 1. How do we figure out the cause of the connect failures?
> 2. What's the controller anyway?
> 3. Are there some command-line diagnostic tools for inspecting the health of the system?
>
> Thanks for any help,
> Chuck
>
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowiecki@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com <ma...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612