You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lianetm (via GitHub)" <gi...@apache.org> on 2025/05/26 11:35:09 UTC
[PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
lianetm opened a new pull request, #19814:
URL: https://github.com/apache/kafka/pull/19814
This PR uses topic IDs received in assignment (under new protocol) to ensure that only these assigned topics are included in the consumer metadata requests performed when the user subscribes to broker-side regex (RE2J)
With the changes we also end up fixing another issue (KAFKA-18729) that aimed to avoid iterating the full set of assigned partitions when checking if a topic should be retained from the metadata response when using RE2J.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #19814:
URL: https://github.com/apache/kafka/pull/19814#discussion_r2132875505
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java:
##########
@@ -66,14 +67,24 @@ public boolean allowAutoTopicCreation() {
return allowAutoTopicCreation;
}
+ /**
+ * Constructs a metadata request builder for fetching cluster metadata for the topics the consumer needs.
+ * This will include:
+ * <ul>
+ * <li>topics the consumer is subscribed to using topic names (calls to subscribe with topic name list or client-side regex)</li>
+ * <li>topics the consumer is subscribed to using topic IDs (calls to subscribe with broker-side regex RE2J)</li>
+ * <li>topics involved in calls for fetching offsets (transient topics)</li>
+ * </ul>
+ * Note that this will generate a request for all topics in the cluster only when the consumer is subscribed to a client-side regex.
+ */
@Override
public synchronized MetadataRequest.Builder newMetadataRequestBuilder() {
- if (subscription.hasPatternSubscription() || subscription.hasRe2JPatternSubscription())
+ if (subscription.hasPatternSubscription())
return MetadataRequest.Builder.allTopics();
List<String> topics = new ArrayList<>();
topics.addAll(subscription.metadataTopics());
topics.addAll(transientTopics);
- return new MetadataRequest.Builder(topics, allowAutoTopicCreation);
+ return new MetadataRequest.Builder(topics, subscription.assignedTopicIds(), allowAutoTopicCreation);
Review Comment:
good catch (mismatch between the javadoc and code because I did add the code at some point but seems I ended up removing it unintentionally).
You're right that we could need metadata for assigned topic IDs and transient topics (names), but since the metadata request handling on the broker does not support that we have no choice but to request them all. Added the fix.
Since we're here, curious, would it make sense to review that broker-side logic to process metadata requests and take both, names and IDs if present? (this bit, that at the moment only considers topic names if no topic IDs provided https://github.com/apache/kafka/blob/c4a769bc8be54fe07e0ed5f05f00b48aa02a8404/core/src/main/scala/kafka/server/KafkaApis.scala#L899-L902)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #19814:
URL: https://github.com/apache/kafka/pull/19814#discussion_r2132881121
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java:
##########
@@ -91,6 +92,13 @@ private enum SubscriptionType {
/* the list of topics the user has requested */
private Set<String> subscription;
+ /**
+ * Topic IDs received in an assignment from the coordinator when using the Consumer rebalance protocol.
+ * This will be used to include assigned topic IDs in metadata requests when the consumer
+ * does not know the topic names (ex. when the user subscribes to a RE2J regex computed on the broker)
+ */
+ private Set<Uuid> assignedTopicIds;
+
Review Comment:
yeap, totally agree, just that I gave it a try and seemed like too much of a refactoring on the `MembershipMgr` and `SubscriptionState` at this point (close to the 4.1 cut), so went with the small fix to ensure it lands.
But totally agree that we could extract the assignment state that is now into the subscription state for the classic consumer, and in the subscriptionState and membershipMgr for the new consumer. Filing a jira and I'll address that myself after this if that's ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #19814:
URL: https://github.com/apache/kafka/pull/19814#discussion_r2132881121
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java:
##########
@@ -91,6 +92,13 @@ private enum SubscriptionType {
/* the list of topics the user has requested */
private Set<String> subscription;
+ /**
+ * Topic IDs received in an assignment from the coordinator when using the Consumer rebalance protocol.
+ * This will be used to include assigned topic IDs in metadata requests when the consumer
+ * does not know the topic names (ex. when the user subscribes to a RE2J regex computed on the broker)
+ */
+ private Set<Uuid> assignedTopicIds;
+
Review Comment:
yeap, totally agree, just that I gave it a try and seemed like too much of a refactoring on the `MembershipMgr` and `SubscriptionState` at this point (close to the 4.1 cut), so went with the small fix to ensure it lands.
But totally agree that we could extract the assignment state that is now into the subscription state for the classic consumer, and in the subscriptionState and membershipMgr for the new consumer. Filing a jira and I'll address that myself after this if that's ok.
-- Update
https://issues.apache.org/jira/browse/KAFKA-19393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm merged PR #19814:
URL: https://github.com/apache/kafka/pull/19814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #19814:
URL: https://github.com/apache/kafka/pull/19814#discussion_r2138798855
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java:
##########
@@ -66,14 +67,24 @@ public boolean allowAutoTopicCreation() {
return allowAutoTopicCreation;
}
+ /**
+ * Constructs a metadata request builder for fetching cluster metadata for the topics the consumer needs.
+ * This will include:
+ * <ul>
+ * <li>topics the consumer is subscribed to using topic names (calls to subscribe with topic name list or client-side regex)</li>
+ * <li>topics the consumer is subscribed to using topic IDs (calls to subscribe with broker-side regex RE2J)</li>
+ * <li>topics involved in calls for fetching offsets (transient topics)</li>
+ * </ul>
+ * Note that this will generate a request for all topics in the cluster only when the consumer is subscribed to a client-side regex.
+ */
@Override
public synchronized MetadataRequest.Builder newMetadataRequestBuilder() {
- if (subscription.hasPatternSubscription() || subscription.hasRe2JPatternSubscription())
+ if (subscription.hasPatternSubscription())
return MetadataRequest.Builder.allTopics();
List<String> topics = new ArrayList<>();
topics.addAll(subscription.metadataTopics());
topics.addAll(transientTopics);
- return new MetadataRequest.Builder(topics, allowAutoTopicCreation);
+ return new MetadataRequest.Builder(topics, subscription.assignedTopicIds(), allowAutoTopicCreation);
Review Comment:
Thinking more about this I simplified it a bit more around the case where the consumer may be using boker-side regex and transient topics. So the consumer would need metadata for topic IDs and names at the same time only if all this happens at the same time:
- call to offsets-related APIs (beginning/endOffsets or offsetsForTimes)
- topic not in metadata cache
- new partition assigned using RE2J (in background received in HB), for topic not in metadata cache
Probably an edge case, but went ahead with the change to simply "pipeline" the requests, resolving the reconciliation first (seemed sensible to prioritize not blocking reconciliation/poll, than a single API call to beginningOffsets/endOffsets/offsetsForTimes, thoughts?)
So with the current change the consumer that needs metadata for a topic ID from RE2J and transient topics will send a first request for the assigned topic ID metadata. Once that resolved, it will send the request for transient topics if any, on a next background poll iteration (background poll -> network client poll -> request metadata).
I'm adding more tests but pushed the change to align
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18117; KAFKA-18729: Use assigned topic IDs to avoid full metadata requests on broker-side regex [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #19814:
URL: https://github.com/apache/kafka/pull/19814#discussion_r2131673167
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java:
##########
@@ -91,6 +92,13 @@ private enum SubscriptionType {
/* the list of topics the user has requested */
private Set<String> subscription;
+ /**
+ * Topic IDs received in an assignment from the coordinator when using the Consumer rebalance protocol.
+ * This will be used to include assigned topic IDs in metadata requests when the consumer
+ * does not know the topic names (ex. when the user subscribes to a RE2J regex computed on the broker)
+ */
+ private Set<Uuid> assignedTopicIds;
+
Review Comment:
I think that this is fine for addressing the bug. However, it would be great if we could try to consolidate it with the assignment in the future. I haven't look into it but it seems feasible to consolidate all the various assignment states that we have (here and in the membership manager) into a single view. Have you considered it?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java:
##########
@@ -490,18 +500,16 @@ public synchronized boolean hasAutoAssignedPartitions() {
|| this.subscriptionType == SubscriptionType.AUTO_TOPICS_SHARE || this.subscriptionType == SubscriptionType.AUTO_PATTERN_RE2J;
}
- public synchronized boolean isAssignedFromRe2j(String topic) {
- if (!hasRe2JPatternSubscription()) {
+ /**
+ * Check if the topic ID has been received in an assignment
+ * from the coordinator after subscribing to a broker-side regex.
+ */
+ public synchronized boolean isAssignedFromRe2j(Uuid topic) {
Review Comment:
nit: topicId?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java:
##########
@@ -66,14 +67,24 @@ public boolean allowAutoTopicCreation() {
return allowAutoTopicCreation;
}
+ /**
+ * Constructs a metadata request builder for fetching cluster metadata for the topics the consumer needs.
+ * This will include:
+ * <ul>
+ * <li>topics the consumer is subscribed to using topic names (calls to subscribe with topic name list or client-side regex)</li>
+ * <li>topics the consumer is subscribed to using topic IDs (calls to subscribe with broker-side regex RE2J)</li>
+ * <li>topics involved in calls for fetching offsets (transient topics)</li>
+ * </ul>
+ * Note that this will generate a request for all topics in the cluster only when the consumer is subscribed to a client-side regex.
+ */
@Override
public synchronized MetadataRequest.Builder newMetadataRequestBuilder() {
- if (subscription.hasPatternSubscription() || subscription.hasRe2JPatternSubscription())
+ if (subscription.hasPatternSubscription())
return MetadataRequest.Builder.allTopics();
List<String> topics = new ArrayList<>();
topics.addAll(subscription.metadataTopics());
topics.addAll(transientTopics);
- return new MetadataRequest.Builder(topics, allowAutoTopicCreation);
+ return new MetadataRequest.Builder(topics, subscription.assignedTopicIds(), allowAutoTopicCreation);
Review Comment:
I am a bit confused here. The javadoc says that the metadata request will include all the topics. However, the builder does not do it internally. Why that? I suppose that we could actually have both transient topics and topic ids at the same time, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org