[jira] [Updated] (KAFKA-12405) Inconsistent broker without active topics still involved in coordinating consumer group

["Alexander (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander updated KAFKA-12405:
------------------------------
    Summary: Inconsistent broker without active topics still involved in coordinating consumer group  (was: Broker without active topics still involved in coordinating consumer group)

> Inconsistent broker without active topics still involved in coordinating consumer group
> ---------------------------------------------------------------------------------------
>
>                 Key: KAFKA-12405
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12405
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 2.3.0
>            Reporter: Alexander
>            Priority: Major
>
> Short description:
>  Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group
>  
> Details:
>  We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
>  * After high disk load broker(2) came out of cluster
>  * Brokers (1), (3) shortly became leaders of it's topics
>  * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to
> "Out of sync replicas". Now it start spamming with
>  "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
>  "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
> Like in https://issues.apache.org/jira/browse/KAFKA-3042
> But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem
> Broker(2) server log:
> {noformat}
> [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)"
> [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
> [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat}
>  The number of log entry "Rebalancing group..." per hour
> {noformat}
> $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done
> server.2021-03-02-10.kafka3 5
> server.2021-03-02-11.kafka3 1
> server.2021-03-02-12.kafka3 3
> server.2021-03-02-10.kafka2 1
> server.2021-03-02-11.kafka2 306
> server.2021-03-02-12.kafka2 452
> server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
> {noformat}
>  
> The questions:
>  * Is it a bug or topicless broker should be involved in coordinating consumer group?
>  * If it's a bug, does this fixed in any later kafka release?
> Thank you.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)