[jira] [Created] (KAFKA-3949) Consumer topic subscription change may be ignored if a rebalance is in progress

["Jason Gustafson (JIRA)" <ji...@apache.org>]

Jason Gustafson created KAFKA-3949:
--------------------------------------

             Summary: Consumer topic subscription change may be ignored if a rebalance is in progress
                 Key: KAFKA-3949
                 URL: https://issues.apache.org/jira/browse/KAFKA-3949
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.10.0.0, 0.9.0.1
            Reporter: Jason Gustafson


The consumer's regex subscription works by matching all topics fetched from a metadata update against the provided pattern. When a new topic is created or an old topic is deleted, we update the list of subscribed topics and request a rebalance by setting the {{needsPartitionAssignment}} flag inside {{SubscriptionState}}. On the next call to {{poll()}}, the consumer will observe the flag and begin the rebalance by sending a JoinGroup. The problem is that it does not account for the fact that a rebalance could already be in progress at the time the metadata is updated. This causes the following sequence:

1. Rebalance begins (needsPartitionAssignment is set True)
2. Metadata max age expires and and update is triggered
3. Update returns and causes a topic subscription change (needsPartitionAssignment set again to True).
4. Rebalance completes (needsPartitionAssignment is set False)

In this situation, we will not request a new rebalance which will prevent us from receiving an assignment from any topics added to the consumer's subscription when the metadata was updated. This state will persist until another event causes the group to rebalance.

A related problem may occur if a rebalance is interrupted with the wakeup() API, and the user calls subscribe(topics) with a change to the subscription set. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)