[jira] [Created] (KAFKA-10056) Consumer metadata may use outdated groupSubscription that doesn't contain newly subscribed topics

["Rajini Sivaram (Jira)" <ji...@apache.org>]

Rajini Sivaram created KAFKA-10056:
--------------------------------------

             Summary: Consumer metadata may use outdated groupSubscription that doesn't contain newly subscribed topics
                 Key: KAFKA-10056
                 URL: https://issues.apache.org/jira/browse/KAFKA-10056
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 2.5.0
            Reporter: Rajini Sivaram
            Assignee: Rajini Sivaram
             Fix For: 2.6.0, 2.5.1


From [~hai_lin] in KAFKA-9181:

I did notice some issue after this patch, here is what I observe.

Consumer metadata might skip first metadata update, cause grouopSubscription is not reset. In my case, the consumer coordinator thread hijack the update by calling newMetadataRequestAndVersion with outdated groupSubscription before joinPrepare() happen. The groupSubscription will get reset later and it will eventually get update later, and this won't be an issue for initial consumer subscribe(since the groupSubscription is empty anyway), but it might happen the following subscribe when groupSubscription is not empty. This will create a discrepancy between subscription and groupSubscription, if any new metadata request happened in between, metadataTopics will return outdated group information. 

 
h4. The happy path
 * Consumer call subscribe > Update {{needUpdated}}, bump up {{requestVersion}} and update {{subscription}} in {{SubscriptionState}} > {{prepareJoin()}} was call in first {{poll()}} to reset {{groupSubscription}} -> next time when metadata update was call and {{metadataTopics()}} returns {{subscription}} since {{groupSubscription}} is empty -> update call issue to broker to fetch partition information for new topic

h4. In our case
 * Consumer call subscribe > Update {{needUpdated}}, bump up {{requestVersion}} and update {{subscription}}(not {{groupSubscription}}) in {{SubscriptionState}} > Consumer Coordinator heartbeat thread call metadata request and {{SubscriptionState}} gave away the current requestVersion and outdated {{groupSubscription}} > making request for metadata update with outdated subscription -> request comes back to client and since {{requestVersion}} is up to latest, it reset {{needUpdated}} flag -> {{joinPrepare()}} called and reset {{groupSubscription}} > no new metadata update request follow cause {{needUpdated}} was reset -> metadata request will happen when {{metadata.max.age}} reaches.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)