[jira] [Created] (KAFKA-9829) Kafka brokers are un-registered on Zookeeper node replacement

["Pradeep (Jira)" <ji...@apache.org>]

Pradeep created KAFKA-9829:
------------------------------

             Summary: Kafka brokers are un-registered on Zookeeper node replacement
                 Key: KAFKA-9829
                 URL: https://issues.apache.org/jira/browse/KAFKA-9829
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.10.2.1
            Reporter: Pradeep


We have a Kafka cluster with 3 nodes connected to a Zookeeper (3.4.14) cluster of 3 nodes in AWS. We make use of the auto-scaling group to provision nodes upon failures. We are seeing an issue where the Kafka brokers are getting un-registered when all the Zookeeper nodes are replaced one after the other. Every Zookeeper node is terminated from AWS console and we wait for a replacement node to be provisioned with Zookeeper initialized before terminating the other node.

On every Zookeeper node replacement, the /broker/ids path show all the Kafka broker ids in the cluster. But only on the final Zookeeper node replacement, the content in /broker/ids become empty.

We are seeing below logs in one of the Zookeeper nodes when all of the original nodes are replaced.

{{2020-03-26 20:29:20,303 [myid:3] - INFO [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - Expiring session 0x10003b973b50016, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,303 [myid:3] - INFO [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - Expiring session 0x10003b973b5000e, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,303 [myid:3] - INFO [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - Expiring session 0x30003a126690002, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - Deleting ephemeral node /brokers/ids/1002 for session 0x10003b973b50016}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - Deleting ephemeral node /brokers/ids/1003 for session 0x10003b973b5000e}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - Deleting ephemeral node /controller for session 0x30003a126690002}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - Deleting ephemeral node /brokers/ids/1001 for session 0x30003a126690002}}

 

I am not sure if the issue is related to KAFKA-5473.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)