You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jason Gustafson (Jira)" <ji...@apache.org> on 2020/04/23 20:38:00 UTC
[jira] [Resolved] (KAFKA-9844) Maximum number of members within a
group is not always enforced due to a race condition in join group
[ https://issues.apache.org/jira/browse/KAFKA-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Gustafson resolved KAFKA-9844.
------------------------------------
Fix Version/s: 2.6.0
Resolution: Fixed
> Maximum number of members within a group is not always enforced due to a race condition in join group
> -----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9844
> URL: https://issues.apache.org/jira/browse/KAFKA-9844
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.5.0
> Reporter: David Jacot
> Assignee: David Jacot
> Priority: Major
> Fix For: 2.6.0
>
>
> While analysing https://issues.apache.org/jira/browse/KAFKA-7965, I found out that the maximum number of members constraints is not always enforced due to a race condition.
> When an unknown member joins the group, the group is automatically created if it does not exist. Then, it proceeds with a unknownJoinGroup. On that path, the limit is not enforced because we assumes that the group is empty as this stage because it did not exist. As the lookup and the creation are not protected by a lock, multiple join requests could end up on that path and thus bypass the enforcement.
> Here is example of the logs captured while troubleshooting KAFKA-7965. The test setups 3 consumers and use a limit of 2. The logs show that the three members were able to join the group without being evicted.
> {noformat}
> [2020-04-05 13:29:03,145] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Discovered group coordinator localhost:36449 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:794)
> [2020-04-05 13:29:03,145] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Discovered group coordinator localhost:36449 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:794)
> [2020-04-05 13:29:03,151] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Discovered group coordinator localhost:36449 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:794)
> [2020-04-05 13:29:03,153] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Attempt to heartbeat failed since member id ConsumerTestConsumer-764a71ea-f9b3-462c-9986-8e6b2530d6e3 is not valid. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1054)
> [2020-04-05 13:29:03,155] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Giving away all assigned partitions as lost since generation has been reset,indicating that consumer is no longer part of the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:670)
> [2020-04-05 13:29:03,155] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Lost previously assigned partitions group-max-size-test-5, group-max-size-test-4 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:314)
> [2020-04-05 13:29:03,156] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,154] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Attempt to heartbeat failed since member id ConsumerTestConsumer-2d2886ad-1244-4ef7-9e07-62282c3547fd is not valid. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1054)
> [2020-04-05 13:29:03,156] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Attempt to heartbeat failed since member id ConsumerTestConsumer-42d0fa9d-cfbb-458f-afe9-99a75fef8e08 is not valid. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1054)
> [2020-04-05 13:29:03,157] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Giving away all assigned partitions as lost since generation has been reset,indicating that consumer is no longer part of the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:670)
> [2020-04-05 13:29:03,158] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Lost previously assigned partitions group-max-size-test-2, group-max-size-test-3 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:314)
> [2020-04-05 13:29:03,158] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,157] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Giving away all assigned partitions as lost since generation has been reset,indicating that consumer is no longer part of the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:670)
> [2020-04-05 13:29:03,159] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Lost previously assigned partitions group-max-size-test-1, group-max-size-test-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:314)
> [2020-04-05 13:29:03,159] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,160] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,161] INFO [GroupCoordinator 2]: Preparing to rebalance group group-max-size-test in state PreparingRebalance with old generation 0 (__consumer_offsets-0) (reason: Adding new member ConsumerTestConsumer-84fd5153-c425-464d-a724-04022a0608f7 with group instanceid None) (kafka.coordinator.group.GroupCoordinator:66)
> [2020-04-05 13:29:03,158] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,160] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:551)
> [2020-04-05 13:29:03,171] INFO [GroupCoordinator 2]: Stabilized group group-max-size-test generation 1 (__consumer_offsets-0) (kafka.coordinator.group.GroupCoordinator:66)
> [2020-04-05 13:29:03,605] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Finished assignment for group at generation 1: {ConsumerTestConsumer-84fd5153-c425-464d-a724-04022a0608f7=Assignment(partitions=[group-max-size-test-0, group-max-size-test-1]), ConsumerTestConsumer-e25aedeb-73fd-4fae-b56c-fa929f11a9df=Assignment(partitions=[group-max-size-test-4, group-max-size-test-5]), ConsumerTestConsumer-8ca065a1-2ce4-44d5-881c-c6f01cb0d110=Assignment(partitions=[group-max-size-test-2, group-max-size-test-3])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:605)
> [2020-04-05 13:29:03,606] INFO [GroupCoordinator 2]: Assignment received from leader for group group-max-size-test for generation 1 (kafka.coordinator.group.GroupCoordinator:66)
> [2020-04-05 13:29:03,610] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:502)
> [2020-04-05 13:29:03,611] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Adding newly assigned partitions: group-max-size-test-1, group-max-size-test-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276)
> [2020-04-05 13:29:03,612] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,612] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,611] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:502)
> [2020-04-05 13:29:03,611] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:502)
> [2020-04-05 13:29:03,614] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Adding newly assigned partitions: group-max-size-test-2, group-max-size-test-3 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276)
> [2020-04-05 13:29:03,614] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Adding newly assigned partitions: group-max-size-test-5, group-max-size-test-4 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276)
> [2020-04-05 13:29:03,616] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-2 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,617] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-3 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,617] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-1 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383)
> [2020-04-05 13:29:03,617] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-5 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,618] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Found no committed offset for partition group-max-size-test-4 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1297)
> [2020-04-05 13:29:03,619] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-3 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383)
> [2020-04-05 13:29:03,619] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-4 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383)
> [2020-04-05 13:29:03,645] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-2 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383)
> [2020-04-05 13:29:03,646] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-0 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383)
> [2020-04-05 13:29:03,651] INFO [Consumer clientId=ConsumerTestConsumer, groupId=group-max-size-test] Resetting offset for partition group-max-size-test-5 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:383){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)