[jira] [Created] (KAFKA-6537) Duplicate consumers after consumer group rebalance

["Michael Golovanov (JIRA)" <ji...@apache.org>]

Michael Golovanov created KAFKA-6537:
----------------------------------------

             Summary: Duplicate consumers after consumer group rebalance
                 Key: KAFKA-6537
                 URL: https://issues.apache.org/jira/browse/KAFKA-6537
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.10.1.1
            Reporter: Michael Golovanov


*_Deployment description_*

Kafka brokers have been deployed on ten (10) nodes. Zookeeper cluster have seven (7) nodes. Nodes of Kafka brokers shared with Zookeeper nodes on bare metal hosts.

Broker/Zookeeper hosts OS is Redhat 7 and JVM version is Java 8. Host names are grid48, grid237 и grid251.

We have one topic with six (6) patitions. Kafka consumers deployed on three (3) hosts. Each host have two (2) consumers. All consumers belong to single group.

 

*_Error description_*

After start all consumers Apache Kafka partitions of topic was balanced evenly.

grid237 owns partitions 0,1 (0 - consumer thread-0, 1, consumer thread-1)
 grid251 owns partitions 2,3 (2 - consumer thread-0, consumer thread-1)
 grid48 owns partitions 4,5 (4- consumer thread-0, 5, consumer thread-1)

After some period of time we see haotic revokes and assigns partitions between brokers and then all partitions in log assigned to one consumer on one node grid48

 

But really all partitions read not only by thread-1 consumer, but thread-0 on grid48. And all messages from topic partitions was duplicate. Consumer thread-0 try to commit message offset and get commit error, thread-1 consumer successfully commit offset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)