You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Alexander (Jira)" <ji...@apache.org> on 2021/03/03 15:26:00 UTC
[jira] [Updated] (KAFKA-12405) Broker without active topics still
involved in coordinating consumer group
[ https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander updated KAFKA-12405:
------------------------------
Description:
Short description:
Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat}
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
> Broker without active topics still involved in coordinating consumer group
> --------------------------------------------------------------------------
>
> Key: KAFKA-12405
> URL: https://issues.apache.org/jira/browse/KAFKA-12405
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.3.0
> Reporter: Alexander
> Priority: Major
>
> Short description:
> Broker without active topics and with inconsistent data still involved in coordinating consumer group. It leads to constant rebalancing consumer group
>
> Details:
> We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
> * After high disk load broker(2) came out of cluster
> * Brokers (1), (3) shortly became leaders of it's topics
> * Broker(2) back to cluster, but have failed to became leader of it's preferred topics due to
> "Out of sync replicas". Now it start spamming with
> "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
> "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
> Like in https://issues.apache.org/jira/browse/KAFKA-3042
> But at the same time we start to observing constant rebalancing of consumer group "mirror_maker_group" and topic reading within this group was completely stucked. Restarting broker(2) fix the problem
> Broker(2) server log:
> {noformat}
> [executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason: error when storing group assignment during SyncGroup)"
> [data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
> [data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Assignment received from leader for group mirror_maker_group for generation 7140{noformat}
> The number of log entry "Rebalancing group..." per hour
> {noformat}
> $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log | wc -l; done
> server.2021-03-02-10.kafka3 5
> server.2021-03-02-11.kafka3 1
> server.2021-03-02-12.kafka3 3
> server.2021-03-02-10.kafka2 1
> server.2021-03-02-11.kafka2 306
> server.2021-03-02-12.kafka2 452
> server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
> {noformat}
>
> The questions:
> * Is it a bug or topicless broker should be involved in coordinating consumer group?
> * If it's a bug, does this fixed in any later kafka release?
> Thank you.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)