You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "chickenchickenlove (via GitHub)" <gi...@apache.org> on 2026/04/12 11:59:57 UTC
[PR] [POC] KAFKA-20171: Kafka Streams use topic ids. (other version) [kafka]
chickenchickenlove opened a new pull request, #22029:
URL: https://github.com/apache/kafka/pull/22029
## Summary
One possible way to support server-side regex in the StreamsGroup Protocol is for the broker to send topic IDs, and for Kafka Streams to resolve those topic IDs back to topic names on the client side before using them.
The ConsumerGroup Protocol already follows a similar model: the broker sends topic IDs, and the client only consumes topics whose names it can resolve locally. This POC explores the same general approach for the StreamsGroup Protocol.
This change introduces the end-to-end plumbing for that model. The broker continues to produce the authoritative task assignment, while the client materializes that assignment into fetchable `TaskId -> TopicPartition` mappings using local metadata.
## Scope
This is still a POC.
The current implementation does not yet include server-side regex source topic support itself. However, the remaining path appears straightforward: the broker would resolve regex source topics and send the corresponding topic IDs to the client, while the client would relax the regex-related constraints accordingly.
Also, broken method signatures in JavaDoc and some test code have not yet been cleaned up in this POC.
## Layering
This implementation can be divided into four layers.
### (1) Broker Assignment Layer
The broker produces the authoritative assignment and includes resolved topic IDs by subtopology in the response. The source of truth at this layer is the full task snapshot.
### (2) Membership / Client Control Plane Layer
StreamsMembershipManager manages the broker assignment in this layer. It operates on the full task snapshot and is responsible for owned-task reporting to the broker as well as callback sequencing.
### (3) Materialization / Fetch Layer
This layer resolves the topic IDs received from the broker into topic names using client metadata, and builds the actual fetchable TaskId -> TopicPartition plan. The source of truth in this layer is the materialized subset, and active tasks in subtopologies with unresolved source topics are excluded here.
### (4) Streams Runtime Layer
Callbacks and task control-plane behavior continue to use the full task snapshot. In contrast, the TopicPartitions used for actual runtime assignment and revocation are based on the materialized subset.
## Current Behavior
In this POC, if even one source topic ID in a given subtopology cannot be resolved to a topic name, the active tasks for that subtopology:
1. are not registered in SubscriptionState as TopicPartitions, and
2. are not included in the Streams runtime active assignment.
Once topic id -> topic name resolution becomes possible, they are included again in the materialization plan during the next maybeReconcile(), and are then reflected in fetch and runtime.
## Intent
The key idea of this POC is to keep the broker authoritative at the task-assignment level, while allowing the client to materialize only the subset that is currently resolvable from metadata.
In the previous version, the response included ResolvedTopicIds as a top-level field, but in this version, Resolved Topic Ids are included within the ActiveTaskIds structure.
- previous version: https://github.com/apache/kafka/pull/21909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [POC] KAFKA-20171: Kafka Streams use topic ids. (other version) [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove closed pull request #22029: [POC] KAFKA-20171: Kafka Streams use topic ids. (other version)
URL: https://github.com/apache/kafka/pull/22029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org