You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "JasonLee (Jira)" <ji...@apache.org> on 2022/07/21 08:08:00 UTC

[jira] [Commented] (FLINK-28595) KafkaSource should not read metadata of unmatched regex topics

    [ https://issues.apache.org/jira/browse/FLINK-28595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569301#comment-17569301 ] 

JasonLee commented on FLINK-28595:
----------------------------------

I think Afzal Mazhar is right that we should put the filter forward and I'm happy to fix it

Can someone assign this to me,thanks

> KafkaSource should not read metadata of unmatched regex topics
> --------------------------------------------------------------
>
>                 Key: FLINK-28595
>                 URL: https://issues.apache.org/jira/browse/FLINK-28595
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>            Reporter: Afzal Mazhar
>            Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When we use a regex to subscribe to topics, the current connector gets a list of all topics, then runs describe against all of them, and finally filters by the regex pattern. This is not performant, as well as could possibly trigger audit alarms against sensitive topics that do not match the regex.
> Proposed fix: move the regex filtering from the TopicPatternSubscriber's set() down into KafkaSubscriberUtils getAllTopicMetadata(). Get the list of topics, filter by pattern (if any), then get metadata. Create appropriate tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)