You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jonathan Santilli (JIRA)" <ji...@apache.org> on 2018/11/10 16:47:00 UTC
[jira] [Updated] (KAFKA-6584) Session expiration concurrent with
ZooKeeper leadership failover may lead to broker registration failure
[ https://issues.apache.org/jira/browse/KAFKA-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Santilli updated KAFKA-6584:
-------------------------------------
Description:
It seems that an edge case exists which can lead to sessions "un-expiring" during a ZooKeeper leadership failover. Additional details can be found in ZOOKEEPER-2985.
This leads to a NODEXISTS error when attempting to re-create the ephemeral brokers/ids/\{id} node in ZkUtils.registerBrokerInZk. We experienced this issue on each node within a 3-node Kafka cluster running 1.0.0. All three nodes continued running (producers and consumers appeared unaffected), but none of the nodes were considered online and partition leadership could be not re-assigned.
I took a quick look at trunk and I believe the issue is still present, but has moved into KafkaZkClient.checkedEphemeralCreate which will [raise an error|https://github.com/apache/kafka/blob/90e0bbe/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512] when it finds that the broker/ids/\{id} node exists, but belongs to the old (believed expired) session.
*NOTE:* KAFKA-7165 introduce a workaround to cope with the case described here. We decided to keep this issue open to track the ZOOKEEPER-2985 status.
was:
It seems that an edge case exists which can lead to sessions "un-expiring" during a ZooKeeper leadership failover. Additional details can be found in ZOOKEEPER-2985.
This leads to a NODEXISTS error when attempting to re-create the ephemeral brokers/ids/\{id} node in ZkUtils.registerBrokerInZk. We experienced this issue on each node within a 3-node Kafka cluster running 1.0.0. All three nodes continued running (producers and consumers appeared unaffected), but none of the nodes were considered online and partition leadership could be not re-assigned.
I took a quick look at trunk and I believe the issue is still present, but has moved into KafkaZkClient.checkedEphemeralCreate which will [raise an error|https://github.com/apache/kafka/blob/90e0bbe/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512] when it finds that the broker/ids/\{id} node exists, but belongs to the old (believed expired) session.
> Session expiration concurrent with ZooKeeper leadership failover may lead to broker registration failure
> --------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-6584
> URL: https://issues.apache.org/jira/browse/KAFKA-6584
> Project: Kafka
> Issue Type: Bug
> Components: zkclient
> Affects Versions: 1.0.0
> Reporter: Chris Thunes
> Priority: Major
>
> It seems that an edge case exists which can lead to sessions "un-expiring" during a ZooKeeper leadership failover. Additional details can be found in ZOOKEEPER-2985.
> This leads to a NODEXISTS error when attempting to re-create the ephemeral brokers/ids/\{id} node in ZkUtils.registerBrokerInZk. We experienced this issue on each node within a 3-node Kafka cluster running 1.0.0. All three nodes continued running (producers and consumers appeared unaffected), but none of the nodes were considered online and partition leadership could be not re-assigned.
> I took a quick look at trunk and I believe the issue is still present, but has moved into KafkaZkClient.checkedEphemeralCreate which will [raise an error|https://github.com/apache/kafka/blob/90e0bbe/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512] when it finds that the broker/ids/\{id} node exists, but belongs to the old (believed expired) session.
>
> *NOTE:* KAFKA-7165 introduce a workaround to cope with the case described here. We decided to keep this issue open to track the ZOOKEEPER-2985 status.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)