You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2025/10/16 16:31:28 UTC
[PR] KAFKA-19779: Introduce source topic to subtopology map [2/N] [kafka]
lucasbru opened a new pull request, #20717:
URL: https://github.com/apache/kafka/pull/20717
For validating offset commits based on partitions, we need to be able to efficiently find the subtopology from a given source topic (user input topic or repartition topic).
We introduce this precomputed map in `StreamsTopology`, where it is generated upon construction.
We can use the precomputed map in StreamsGroup.isSuscribedToTopic we can use it to validate efficiently if we are still subscribed to a certain topic.
Using `StreamTopology` here instead of `ConfiguredTopology` is an improvement in itself, because configured topology is soft state and not entirely reliable to always be there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19779: Introduce source topic to subtopology map [2/N] [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #20717:
URL: https://github.com/apache/kafka/pull/20717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19779: Introduce source topic to subtopology map [2/N] [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20717:
URL: https://github.com/apache/kafka/pull/20717#discussion_r2437904535
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/StreamsTopology.java:
##########
@@ -41,15 +42,45 @@
* @param topologyEpoch The epoch of the topology (must be non-negative).
* @param subtopologies The subtopologies of the topology containing information about source topics,
* repartition topics, changelog topics, co-partition groups etc. (must be non-null)
+ * @param sourceTopicMap A precomputed map of source topics to their corresponding subtopology (must be non-null)
*/
public record StreamsTopology(int topologyEpoch,
- Map<String, Subtopology> subtopologies) {
+ Map<String, Subtopology> subtopologies,
+ Map<String, Subtopology> sourceTopicMap) {
+
+ /**
+ * Constructor that automatically computes the sourceTopicMap from subtopologies.
+ *
+ * @param topologyEpoch The epoch of the topology (must be non-negative).
+ * @param subtopologies The subtopologies of the topology.
+ */
+ public StreamsTopology(int topologyEpoch, Map<String, Subtopology> subtopologies) {
+ this(topologyEpoch, subtopologies, computeSourceTopicMap(subtopologies));
+ }
public StreamsTopology {
if (topologyEpoch < 0) {
throw new IllegalArgumentException("Topology epoch must be non-negative.");
}
subtopologies = Collections.unmodifiableMap(Objects.requireNonNull(subtopologies, "Subtopologies cannot be null."));
+ sourceTopicMap = Collections.unmodifiableMap(Objects.requireNonNull(sourceTopicMap, "Source topic map cannot be null."));
+ }
+
+ private static Map<String, Subtopology> computeSourceTopicMap(Map<String, Subtopology> subtopologies) {
+ Objects.requireNonNull(subtopologies, "Subtopologies cannot be null.");
+ Map<String, Subtopology> computedMap = new HashMap<>();
+ for (Map.Entry<String, Subtopology> entry : subtopologies.entrySet()) {
+ Subtopology subtopology = entry.getValue();
Review Comment:
Seems we only access the values? Should we simplify the code to:
```
for (Subtopology subtopology : subtopologies.values()) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org