You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Christine Wright (JIRA)" <ji...@apache.org> on 2017/07/12 17:50:00 UTC
[jira] [Commented] (KAFKA-4277) creating ephemeral node already
exist
[ https://issues.apache.org/jira/browse/KAFKA-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084382#comment-16084382 ]
Christine Wright commented on KAFKA-4277:
-----------------------------------------
I have also seen this issue,
zookeeper 3.4.6 cluster of 3
kafka_2.11-0.10.1.1 cluster of 5
I have full zook and kafka logs, but I'm not sure they will provide more information that what was provided above.
There was a period of network instability with lots of zookeeper timeouts and reconnects on the entire kafka cluster, but all was fine until one broker hit this problem
ERROR Error handling event ZkEvent[New session event sent to kafka.server.KafkaHealthcheck$SessionExpireListener@2a31b49a] (org.I0Itec.zkclient.ZkEventThread)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/4. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:393)
and on zookeeper I see:
2017-07-08 00:02:24,478 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x6 zxid:0x900000012 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NodeExists for /brokers
2017-07-08 00:02:24,478 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x7 zxid:0x900000013 txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists for /brokers/ids
2017-07-08 00:02:24,478 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x8 zxid:0x900000014 txntype:-1 reqpath:n/a Error Path:/brokers/ids/4 Error:KeeperErrorCode = NodeExists for /brokers/ids/4
> creating ephemeral node already exist
> -------------------------------------
>
> Key: KAFKA-4277
> URL: https://issues.apache.org/jira/browse/KAFKA-4277
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.0.0
> Reporter: Feixiang Yan
>
> I use zookeeper 3.4.6.
> Zookeeper session time out, zkClient try reconnect failed. Then re-establish the session and re-registering broker info in ZK, throws NODEEXISTS Exception.
> I think it is because the ephemeral node which created by old session has not removed.
> I read the [ZkUtils.scala|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/utils/ZkUtils.scala] of 0.8.1, createEphemeralPathExpectConflictHandleZKBug try create node in a while loop until create success. This can solve the issue. But in [ZkUtils.scala|https://github.com/apache/kafka/blob/0.10.0.1/core/src/main/scala/kafka/utils/ZkUtils.scala] 0.10.1 the function removed.
> {noformat}
> [2016-10-07 19:00:32,562] INFO Socket connection established to 10.191.155.238/10.191.155.238:21819, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,563] INFO zookeeper state changed (Expired) (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,564] INFO Unable to reconnect to ZooKeeper service, session 0x1576b11f9b201bd has expired, closing socket connection (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,564] INFO Initiating client connection, connectString=10.191.155.237:21819,10.191.155.238:21819,10.191.155.239:21819/cluster2 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@ae71be2 (org.apache.zookeeper.ZooKeeper)
> [2016-10-07 19:00:32,566] INFO Opening socket connection to server 10.191.155.237/10.191.155.237:21819. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO Socket connection established to 10.191.155.237/10.191.155.237:21819, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO Session establishment complete on server 10.191.155.237/10.191.155.237:21819, sessionid = 0x1579ecd39c20006, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,608] INFO re-registering broker info in ZK for broker 3 (kafka.server.KafkaHealthcheck$SessionExpireListener)
> [2016-10-07 19:00:32,610] INFO Creating /brokers/ids/3 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,611] INFO Result of znode creation is: NODEEXISTS (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,614] ERROR Error handling event ZkEvent[New session event sent to kafka.server.KafkaHealthcheck$SessionExpireListener@324f1bc] (org.I0Itec.zkclient.ZkEventThread)
> java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/3. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
> at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
> at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
> at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
> at kafka.server.KafkaHealthcheck$SessionExpireListener.handleNewSession(KafkaHealthcheck.scala:104)
> at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:735)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)