You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/07/21 09:00:46 UTC
[GitHub] [flink] JasonLeeCoding opened a new pull request, #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
JasonLeeCoding opened a new pull request, #20327:
URL: https://github.com/apache/flink/pull/20327
## What is the purpose of the change
*KafkaSource should not read metadata of unmatched regex topics, Optimize the logic for filtering topics*
## Brief change log
*The filter logic from TopicPatternSubscriber# getSubscribedTopicPartitions method moves to KafkaSubscriberUtils# getAllTopicMetadata method*
## Verifying this change
This change is already covered by existing tests, such as *KafkaSubscriberTest#testTopicPatternSubscriber*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1191497368
> We should support the case where no Pattern exists and a caller needs the entire list of topics with metadata.
hi @amazhar1 ,Thanks for your comment, when Pattern is not exists, user should use setTopics method, the TopicListSubscriber object will be called at the bottom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1209135629
@PatrickRen hi, please help to review this PR, When you have free time.thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1202515113
This change is already covered by existing tests, you can refer to KafkaSubscriberTest#testTopicPatternSubscriber.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] amazhar1 commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
amazhar1 commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1192553353
I like it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] amazhar1 commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
amazhar1 commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1201197293
One alternative is to move this function back into KafkaSubscriberUtils so that both the unfiltered and this new filtered method is available for all (future) callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1758314627
@JasonLeeCoding The Flink Kafka connector resides in it's own repository, if this code change is still relevant, please open the PR in https://github.com/apache/flink-connector-kafka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] amazhar1 commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
amazhar1 commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1191460871
We should support the case where no Pattern exists and a caller needs the entire list of topics with metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1201968764
That makes sense. It would be friendlier. I've changed it that way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1191517673
> I was thinking that KafkaSubscriberUtils is a general purpose utility class that any other code can call. While today it is only invoked by the Regex subscriber, tomorrow something else might need a list of topics, with or without a regex filter.
yeah, u are right, maybe we can move getAllTopicMetadata method to TopicPatternSubscriber object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] amazhar1 commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
amazhar1 commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1202494386
Thank you very much for doing this Jason. Do you think you could add a unit test for this method to make it easier for the assigned reviewer to review/merge things? I have not looked but there must be a mock for the adminclient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser closed pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
URL: https://github.com/apache/flink/pull/20327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1191244256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "74eb5dca1c753529e18903f48173b163c11949d1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "74eb5dca1c753529e18903f48173b163c11949d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74eb5dca1c753529e18903f48173b163c11949d1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] amazhar1 commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
amazhar1 commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1191508775
I was thinking that KafkaSubscriberUtils is a general purpose utility class that any other code can call. While today it is only invoked by the Regex subscriber, tomorrow something else might need a list of topics, with or without a regex filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1205184074
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #20327: [FLINK-28595][Connectors/kafka] KafkaSource should not read metadata of unmatched regex topics
Posted by GitBox <gi...@apache.org>.
JasonLeeCoding commented on PR #20327:
URL: https://github.com/apache/flink/pull/20327#issuecomment-1204686180
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org