[jira] [Created] (KAFKA-13543) Consumer may pass stale cluster metadata to the assignor following a subscription update

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

A. Sophie Blee-Goldman created KAFKA-13543:
----------------------------------------------

             Summary: Consumer may pass stale cluster metadata to the assignor following a subscription update
                 Key: KAFKA-13543
                 URL: https://issues.apache.org/jira/browse/KAFKA-13543
             Project: Kafka
          Issue Type: Bug
          Components: consumer
            Reporter: A. Sophie Blee-Goldman


A consumer only ever tracks metadata corresponding to its subscribed topics, which can cause a race condition during a rebalance immediately after a change to the consumer's subscription. Particularly, when new topics are added to the subscription but a rebalance in kicked off before the consumer's metadata is updated with the new topics, it will pass a stale copy of the cluster metadata in to the ConsumerPartitionAssignor#assign method, which may not include the newly subscribed topics regardless of whether they do or do not exist.

Most apps are likely unaffected by this, including any consumer client apps using OOTB assignors, since a new rebalance will be kicked off when the metadata is updated and any partitions from the new topics will be assigned at that time. But in Kafka Streams, we do a check during each rebalance to ensure that any user input topics are created ahead of time. This race condition can result in Streams incorrectly identifying user topics as missing and throwing a MissingSourceTopicException when a new topology subscribed to new topics is added to the application 

We can work around this for now, but it's unfortunate that we can't distinguish between true missing source topics and a transient lack of these topics in the metadata. There might also be some plain consumer client apps with custom assignors that run into this as well, for more advanced users.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)