You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by 徐朝晖 <xu...@163.com> on 2018/05/11 12:13:55 UTC
client cannot connect to zookeeper after node replacement
Hi there,
We were recently running into an issue in cloud env.
Specifically, we have a kafka cluster setup in cloud, and it turns out to be a failure when the zookeeper node is replaced, namely the ip changed while the hostname remains same. We have patched PR150 (https://github.com/apache/zookeeper/pull/150/files) but it seems still not works. While the new zk node ip can be re-resolved and zk client in kafka can establish connection to zk node, but the client seems not able to read content from the connection. Anyone encounters similar issue?
Pasted the log from kafka server.
[2018-05-11 03:47:46,806] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] INFO Opening socket connection to server host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name (org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:50,667] WARN Cannot re-resolve server: host.name/10.147.164.83:2181 (org.apache.zookeeper.client.StaticHostProvider)
java.net.UnknownHostException: host.name
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:142)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[2018-05-11 03:47:51,668] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] INFO Opening socket connection to server host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name (org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:54,429] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] INFO Opening socket connection to server host.name/10.147.161.203:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,383] INFO Socket connection established to host.name/10.147.161.203:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,386] INFO Unable to read additional data from server sessionid 0x1634d42d88a001e, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
this ends up with continuous log flushing, the log is repeated in a sequence like above highlighted ones.
Anyone can share thoughts regarding this issue?
Thanks,
Jeff
Re: client cannot connect to zookeeper after node replacement
Posted by Andor Molnar <an...@cloudera.com>.
Hi Jeff,
There's another patch which is supposed to be the successor of #151 here:
https://github.com/apache/zookeeper/pull/451
Would you please try this one too and make sure it fixes the problem?
Thanks,
Andor
On Fri, May 11, 2018 at 5:13 AM, 徐朝晖 <xu...@163.com> wrote:
> Hi there,
>
>
>
> We were recently running into an issue in cloud env.
> Specifically, we have a kafka cluster setup in cloud, and it turns out to
> be a failure when the zookeeper node is replaced, namely the ip changed
> while the hostname remains same. We have patched PR150 (
> https://github.com/apache/zookeeper/pull/150/files) but it seems still
> not works. While the new zk node ip can be re-resolved and zk client in
> kafka can establish connection to zk node, but the client seems not able to
> read content from the connection. Anyone encounters similar issue?
>
> Pasted the log from kafka server.
>
> [2018-05-11 03:47:46,806] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:46,806] INFO Opening socket connection to server
> host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
> [2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null,
> unexpected error, closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> [2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name
> (org.apache.zookeeper.client.StaticHostProvider)
> [2018-05-11 03:47:50,667] WARN Cannot re-resolve server:
> host.name/10.147.164.83:2181 (org.apache.zookeeper.client.
> StaticHostProvider)
> java.net.UnknownHostException: host.name
> at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at org.apache.zookeeper.client.StaticHostProvider.next(
> StaticHostProvider.java:142)
> at org.apache.zookeeper.ClientCnxn$SendThread.
> startConnect(ClientCnxn.java:997)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
> [2018-05-11 03:47:51,668] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:51,668] INFO Opening socket connection to server
> host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
> [2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null,
> unexpected error, closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> [2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name
> (org.apache.zookeeper.client.StaticHostProvider)
> [2018-05-11 03:47:54,429] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:54,429] INFO Opening socket connection to server
> host.name/10.147.161.203:2181 (org.apache.zookeeper.ClientCnxn)
> [2018-05-11 03:47:55,383] INFO Socket connection established to
> host.name/10.147.161.203:2181, initiating session (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:55,386] INFO Unable to read additional data from server
> sessionid 0x1634d42d88a001e, likely server has closed socket, closing
> socket connection and attempting reconnect (org.apache.zookeeper.
> ClientCnxn)
>
>
>
>
> this ends up with continuous log flushing, the log is repeated in a
> sequence like above highlighted ones.
>
>
>
>
> Anyone can share thoughts regarding this issue?
>
>
>
>
> Thanks,
>
> Jeff
>
>
>
>
>
>