[jira] [Commented] (KAFKA-13543) Consumer may pass stale cluster metadata to the assignor following a subscription update

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458923#comment-17458923 ] 

A. Sophie Blee-Goldman commented on KAFKA-13543:
------------------------------------------------

One possible fix would be to block inside Consumer#subscribe until the metadata has been updated, as we currently just issue a non-blocking call to update the metadata and then return. Of course there could be some unintended repercussions to making this previously async API suddenly a blocking call, so we should definitely discuss that further.

If waiting on the metadata update is off the table another option is to just mitigate the issue by at least giving the assignor a way to tell that the metadata has not been updated since the subscription was, and therefore that the cluster metadata passed in to #assign can not necessarily be trusted. 

> Consumer may pass stale cluster metadata to the assignor following a subscription update
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13543
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13543
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: A. Sophie Blee-Goldman
>            Priority: Major
>
> A consumer only ever tracks metadata corresponding to its subscribed topics, which can cause a race condition during a rebalance immediately after a change to the consumer's subscription. Particularly, when new topics are added to the subscription but a rebalance in kicked off before the consumer's metadata is updated with the new topics, it will pass a stale copy of the cluster metadata in to the ConsumerPartitionAssignor#assign method, which may not include the newly subscribed topics regardless of whether they do or do not exist.
> Most apps are likely unaffected by this, including any consumer client apps using OOTB assignors, since a new rebalance will be kicked off when the metadata is updated and any partitions from the new topics will be assigned at that time. But in Kafka Streams, we do a check during each rebalance to ensure that any user input topics are created ahead of time. This race condition can result in Streams incorrectly identifying user topics as missing and throwing a MissingSourceTopicException when a new topology subscribed to new topics is added to the application 
> We can work around this for now, but it's unfortunate that we can't distinguish between true missing source topics and a transient lack of these topics in the metadata. There might also be some plain consumer client apps with custom assignors that run into this as well, for more advanced users.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)