[jira] [Commented] (KAFKA-18690) New consumer doesn't updateLatestMetadata for topics matching new regex

["Kirk True (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922966#comment-17922966 ] 

Kirk True commented on KAFKA-18690:
-----------------------------------

[~lianetm]—is this a client fix or broker?

> New consumer doesn't updateLatestMetadata for topics matching new regex
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-18690
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18690
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Lianet Magrans
>            Assignee: Sean Quah
>            Priority: Blocker
>              Labels: kip-848-client-support
>             Fix For: 4.0.0
>
>
> When the consumer subscribes to the new RE2J regex resolved on the broker, it's not keeping the leader and epoch info from the metadata response, so not able to properly update the fetch positions.
> After getting the assignment with the topics matching the regex, and receiving the topics in a metadata response, the consumer is not retaining topics that are assigned with Re2Jpattern so fails with 
> Leader for partition {} is unknown for fetching offset {}
>  
> This seems to be the reason behind it, not retaining topics assigned with RE2J
> [https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java#L97 |https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java#L97]
> So when handling the metadata response that includes the matching topics it will never make it to update the leader/epoch info.
> [https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L521]
> To validate with should end with a test similar to [https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/core/src/test/scala/integration/kafka/api/PlaintextConsumerSubscriptionTest.scala#L251] but including consumption.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)