You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2026/06/01 18:43:58 UTC

[PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

dajac opened a new pull request, #22443:
URL: https://github.com/apache/kafka/pull/22443

   This patch lets `TxnOffsetCommit` requests negotiate v6+ (KIP-1319)
   when the producer's metadata cache knows the topic ids for every
   topic in the offsets map.
   
   Key changes:
   - `TransactionManager` gains a `Metadata` reference (constructor
     parameter wired from `KafkaProducer`). It is used to look up topic
     ids when building a `TxnOffsetCommit` request.
   - `txnOffsetCommitHandler` makes a single pass over the offsets map:
     it populates `pendingTxnOffsetCommits`, builds each request topic
     (via `computeIfAbsent` over a name -> topic map, with the topic id
     resolved from `metadata.topicIds()`), and accumulates a snapshot
     `Map<Uuid, String> topicNamesByIds` for the response handler. When
     every topic has a non-zero id, the builder is chosen via
     `forTopicIdsOrNames(...)` so v6 can be negotiated; otherwise it
     falls back to `forTopicNames(...)` and caps at v5.
   - The response handler resolves the topic name from the snapshot
     (not the live metadata cache) when a v6+ response topic omits the
     name on the wire. If the id is unknown to the snapshot, the
     handler logs a warning and skips that topic so the retry loop is
     unaffected.
   
   Test changes:
   - `prepareTxnOffsetCommitResponse` in `TransactionManagerTest` now
     asserts that the inspected request is at version < 6 and that
     every request topic carries a non-empty name. This strengthens the
     existing v5-path coverage.
   - Three new tests cover the v6 path. They seed the metadata cache
     with a known topic id via `RequestTestUtils.metadataUpdateWithIds`
     and then assert the request's negotiated version, topic ids, and
     error handling: `testTxnOffsetCommitNegotiatesV6WhenAllTopicIdsAreAvailable`,
     `testTxnOffsetCommitDowngradesToV5WhenAnyTopicIdIsMissing`, and
     `testTxnOffsetCommitRetriesOnUnknownTopicIdAtV6`.
   
   `KafkaProducer.sendOffsetsToTransaction` will be updated in a
   follow-up to refresh the metadata cache before delegating, so the
   v6 negotiation actually fires for topics that the producer has not
   seen yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3336966568


##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
             int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
 
             this.apiVersions = apiVersions;
+            List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);

Review Comment:
   uh we're changing this in another PR btw, for KIP-909..this one will go first I imagine,  so we need to align on the other one. cc @frankvicky



##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
     private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
                                                           Map<TopicPartition, OffsetAndMetadata> offsets,
                                                           ConsumerGroupMetadata groupMetadata) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
-                    offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
-            pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+        // Resolve topic ids from the metadata cache at request build time.
+        // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+        // is fresh for these topics, so this is a non-blocking lookup.
+        var topicIds = metadata.topicIds();

Review Comment:
   Thinking about the case of Unknown_topic_ID made me notice, we have the metadata check here only (when creating the `txnOffsetCommitHandler`) , but all the retry logic relies on re-enqueueing the same handler, so this won't be triggered on retries I expect (would re-send the same topic-id if we get unknown_topic_id).  
   
   Is the intention to handle that when processing the Unknown_topic_id? (separate PR?)



##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
     private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
                                                           Map<TopicPartition, OffsetAndMetadata> offsets,
                                                           ConsumerGroupMetadata groupMetadata) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
-                    offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
-            pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+        // Resolve topic ids from the metadata cache at request build time.
+        // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+        // is fresh for these topics, so this is a non-blocking lookup.
+        var topicIds = metadata.topicIds();
+        var requestTopicsByName = new HashMap<String, TxnOffsetCommitRequestTopic>();
+        var topicNamesByIds = new HashMap<Uuid, String>();
+        var topics = new ArrayList<TxnOffsetCommitRequestTopic>();
+        var allHaveTopicIds = true;
+        for (var entry : offsets.entrySet()) {
+            var tp = entry.getKey();
+            var offsetAndMetadata = entry.getValue();
+            pendingTxnOffsetCommits.put(
+                tp,
+                new CommittedOffset(
+                    offsetAndMetadata.offset(),
+                    offsetAndMetadata.metadata(),
+                    offsetAndMetadata.leaderEpoch()
+                )
+            );
+            var topicId = topicIds.getOrDefault(tp.topic(), Uuid.ZERO_UUID);
+            allHaveTopicIds &= !topicId.equals(Uuid.ZERO_UUID);
+            var topic = requestTopicsByName.computeIfAbsent(tp.topic(), name -> {
+                var t = new TxnOffsetCommitRequestTopic().setName(name).setTopicId(topicId);
+                topics.add(t);
+                if (!topicId.equals(Uuid.ZERO_UUID)) {
+                    topicNamesByIds.put(topicId, name);
+                }
+                return t;
+            });
+            topic.partitions().add(new TxnOffsetCommitRequestData.TxnOffsetCommitRequestPartition()
+                .setPartitionIndex(tp.partition())
+                .setCommittedOffset(offsetAndMetadata.offset())
+                .setCommittedLeaderEpoch(offsetAndMetadata.leaderEpoch().orElse(RecordBatch.NO_PARTITION_LEADER_EPOCH))
+                .setCommittedMetadata(offsetAndMetadata.metadata()));
         }
 
-        final TxnOffsetCommitRequestData data = new TxnOffsetCommitRequestData()
+        var data = new TxnOffsetCommitRequestData()
             .setTransactionalId(transactionalId)
             .setGroupId(groupMetadata.groupId())
             .setProducerId(producerIdAndEpoch.producerId)
             .setProducerEpoch(producerIdAndEpoch.epoch)
             .setMemberId(groupMetadata.memberId())
             .setGenerationIdOrMemberEpoch(groupMetadata.generationId())
             .setGroupInstanceId(groupMetadata.groupInstanceId().orElse(null))
-            .setTopics(TxnOffsetCommitRequest.getTopics(pendingTxnOffsetCommits));
-        final TxnOffsetCommitRequest.Builder builder =
-            TxnOffsetCommitRequest.Builder.forTopicNames(data, isTransactionV2Enabled());
+            .setTopics(topics);
+        var builder = allHaveTopicIds
+            ? TxnOffsetCommitRequest.Builder.forTopicIdsOrNames(data, isTransactionV2Enabled(), true)

Review Comment:
   this opens the door for the Unknown_topic_id in the response. Is that coming in a separate PR? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #22443:
URL: https://github.com/apache/kafka/pull/22443


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3341874276


##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
     private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
                                                           Map<TopicPartition, OffsetAndMetadata> offsets,
                                                           ConsumerGroupMetadata groupMetadata) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
-                    offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
-            pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+        // Resolve topic ids from the metadata cache at request build time.
+        // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+        // is fresh for these topics, so this is a non-blocking lookup.
+        var topicIds = metadata.topicIds();

Review Comment:
   agreed, makes sense (I was thinking about case 2, deleted)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3339344581


##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
     private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
                                                           Map<TopicPartition, OffsetAndMetadata> offsets,
                                                           ConsumerGroupMetadata groupMetadata) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
-                    offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
-            pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+        // Resolve topic ids from the metadata cache at request build time.
+        // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+        // is fresh for these topics, so this is a non-blocking lookup.
+        var topicIds = metadata.topicIds();
+        var requestTopicsByName = new HashMap<String, TxnOffsetCommitRequestTopic>();
+        var topicNamesByIds = new HashMap<Uuid, String>();
+        var topics = new ArrayList<TxnOffsetCommitRequestTopic>();
+        var allHaveTopicIds = true;
+        for (var entry : offsets.entrySet()) {
+            var tp = entry.getKey();
+            var offsetAndMetadata = entry.getValue();
+            pendingTxnOffsetCommits.put(
+                tp,
+                new CommittedOffset(
+                    offsetAndMetadata.offset(),
+                    offsetAndMetadata.metadata(),
+                    offsetAndMetadata.leaderEpoch()
+                )
+            );
+            var topicId = topicIds.getOrDefault(tp.topic(), Uuid.ZERO_UUID);
+            allHaveTopicIds &= !topicId.equals(Uuid.ZERO_UUID);
+            var topic = requestTopicsByName.computeIfAbsent(tp.topic(), name -> {
+                var t = new TxnOffsetCommitRequestTopic().setName(name).setTopicId(topicId);
+                topics.add(t);
+                if (!topicId.equals(Uuid.ZERO_UUID)) {
+                    topicNamesByIds.put(topicId, name);
+                }
+                return t;
+            });
+            topic.partitions().add(new TxnOffsetCommitRequestData.TxnOffsetCommitRequestPartition()
+                .setPartitionIndex(tp.partition())
+                .setCommittedOffset(offsetAndMetadata.offset())
+                .setCommittedLeaderEpoch(offsetAndMetadata.leaderEpoch().orElse(RecordBatch.NO_PARTITION_LEADER_EPOCH))
+                .setCommittedMetadata(offsetAndMetadata.metadata()));
         }
 
-        final TxnOffsetCommitRequestData data = new TxnOffsetCommitRequestData()
+        var data = new TxnOffsetCommitRequestData()
             .setTransactionalId(transactionalId)
             .setGroupId(groupMetadata.groupId())
             .setProducerId(producerIdAndEpoch.producerId)
             .setProducerEpoch(producerIdAndEpoch.epoch)
             .setMemberId(groupMetadata.memberId())
             .setGenerationIdOrMemberEpoch(groupMetadata.generationId())
             .setGroupInstanceId(groupMetadata.groupInstanceId().orElse(null))
-            .setTopics(TxnOffsetCommitRequest.getTopics(pendingTxnOffsetCommits));
-        final TxnOffsetCommitRequest.Builder builder =
-            TxnOffsetCommitRequest.Builder.forTopicNames(data, isTransactionV2Enabled());
+            .setTopics(topics);
+        var builder = allHaveTopicIds
+            ? TxnOffsetCommitRequest.Builder.forTopicIdsOrNames(data, isTransactionV2Enabled(), true)

Review Comment:
   it is already handled by `error.exception() instanceof RetriableException` as `UNKNOWN_TOPIC_ID` is a `RetriableException` exception. there is also a test covering it: `testTxnOffsetCommitRetriesOnUnknownTopicIdAtV6`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3341871997


##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
             int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
 
             this.apiVersions = apiVersions;
+            List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);

Review Comment:
   not here I would say, we'll handle it in the other PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3337192221


##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
     private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
                                                           Map<TopicPartition, OffsetAndMetadata> offsets,
                                                           ConsumerGroupMetadata groupMetadata) {
-        for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
-            OffsetAndMetadata offsetAndMetadata = entry.getValue();
-            CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
-                    offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
-            pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+        // Resolve topic ids from the metadata cache at request build time.
+        // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+        // is fresh for these topics, so this is a non-blocking lookup.
+        var topicIds = metadata.topicIds();

Review Comment:
   Thinking about the case of Unknown_topic_ID made me notice, we have the metadata check here only (when creating the `txnOffsetCommitHandler`) , but all the retry logic relies on re-enqueueing the same handler, so this won't be triggered on retries I expect (would re-send the same topic-id if we get unknown_topic_id?).  
   
   Is the intention to handle that when processing the Unknown_topic_id? (separate PR?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3339340630


##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
             int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
 
             this.apiVersions = apiVersions;
+            List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);

Review Comment:
   ack. let me know if i need to change anything here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org