You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2026/06/01 18:43:58 UTC
[PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
dajac opened a new pull request, #22443:
URL: https://github.com/apache/kafka/pull/22443
This patch lets `TxnOffsetCommit` requests negotiate v6+ (KIP-1319)
when the producer's metadata cache knows the topic ids for every
topic in the offsets map.
Key changes:
- `TransactionManager` gains a `Metadata` reference (constructor
parameter wired from `KafkaProducer`). It is used to look up topic
ids when building a `TxnOffsetCommit` request.
- `txnOffsetCommitHandler` makes a single pass over the offsets map:
it populates `pendingTxnOffsetCommits`, builds each request topic
(via `computeIfAbsent` over a name -> topic map, with the topic id
resolved from `metadata.topicIds()`), and accumulates a snapshot
`Map<Uuid, String> topicNamesByIds` for the response handler. When
every topic has a non-zero id, the builder is chosen via
`forTopicIdsOrNames(...)` so v6 can be negotiated; otherwise it
falls back to `forTopicNames(...)` and caps at v5.
- The response handler resolves the topic name from the snapshot
(not the live metadata cache) when a v6+ response topic omits the
name on the wire. If the id is unknown to the snapshot, the
handler logs a warning and skips that topic so the retry loop is
unaffected.
Test changes:
- `prepareTxnOffsetCommitResponse` in `TransactionManagerTest` now
asserts that the inspected request is at version < 6 and that
every request topic carries a non-empty name. This strengthens the
existing v5-path coverage.
- Three new tests cover the v6 path. They seed the metadata cache
with a known topic id via `RequestTestUtils.metadataUpdateWithIds`
and then assert the request's negotiated version, topic ids, and
error handling: `testTxnOffsetCommitNegotiatesV6WhenAllTopicIdsAreAvailable`,
`testTxnOffsetCommitDowngradesToV5WhenAnyTopicIdIsMissing`, and
`testTxnOffsetCommitRetriesOnUnknownTopicIdAtV6`.
`KafkaProducer.sendOffsetsToTransaction` will be updated in a
follow-up to refresh the metadata cache before delegating, so the
v6 negotiation actually fires for topics that the producer has not
seen yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3336966568
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = apiVersions;
+ List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);
Review Comment:
uh we're changing this in another PR btw, for KIP-909..this one will go first I imagine, so we need to align on the other one. cc @frankvicky
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition, OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
Review Comment:
Thinking about the case of Unknown_topic_ID made me notice, we have the metadata check here only (when creating the `txnOffsetCommitHandler`) , but all the retry logic relies on re-enqueueing the same handler, so this won't be triggered on retries I expect (would re-send the same topic-id if we get unknown_topic_id).
Is the intention to handle that when processing the Unknown_topic_id? (separate PR?)
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition, OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
+ var requestTopicsByName = new HashMap<String, TxnOffsetCommitRequestTopic>();
+ var topicNamesByIds = new HashMap<Uuid, String>();
+ var topics = new ArrayList<TxnOffsetCommitRequestTopic>();
+ var allHaveTopicIds = true;
+ for (var entry : offsets.entrySet()) {
+ var tp = entry.getKey();
+ var offsetAndMetadata = entry.getValue();
+ pendingTxnOffsetCommits.put(
+ tp,
+ new CommittedOffset(
+ offsetAndMetadata.offset(),
+ offsetAndMetadata.metadata(),
+ offsetAndMetadata.leaderEpoch()
+ )
+ );
+ var topicId = topicIds.getOrDefault(tp.topic(), Uuid.ZERO_UUID);
+ allHaveTopicIds &= !topicId.equals(Uuid.ZERO_UUID);
+ var topic = requestTopicsByName.computeIfAbsent(tp.topic(), name -> {
+ var t = new TxnOffsetCommitRequestTopic().setName(name).setTopicId(topicId);
+ topics.add(t);
+ if (!topicId.equals(Uuid.ZERO_UUID)) {
+ topicNamesByIds.put(topicId, name);
+ }
+ return t;
+ });
+ topic.partitions().add(new TxnOffsetCommitRequestData.TxnOffsetCommitRequestPartition()
+ .setPartitionIndex(tp.partition())
+ .setCommittedOffset(offsetAndMetadata.offset())
+ .setCommittedLeaderEpoch(offsetAndMetadata.leaderEpoch().orElse(RecordBatch.NO_PARTITION_LEADER_EPOCH))
+ .setCommittedMetadata(offsetAndMetadata.metadata()));
}
- final TxnOffsetCommitRequestData data = new TxnOffsetCommitRequestData()
+ var data = new TxnOffsetCommitRequestData()
.setTransactionalId(transactionalId)
.setGroupId(groupMetadata.groupId())
.setProducerId(producerIdAndEpoch.producerId)
.setProducerEpoch(producerIdAndEpoch.epoch)
.setMemberId(groupMetadata.memberId())
.setGenerationIdOrMemberEpoch(groupMetadata.generationId())
.setGroupInstanceId(groupMetadata.groupInstanceId().orElse(null))
- .setTopics(TxnOffsetCommitRequest.getTopics(pendingTxnOffsetCommits));
- final TxnOffsetCommitRequest.Builder builder =
- TxnOffsetCommitRequest.Builder.forTopicNames(data, isTransactionV2Enabled());
+ .setTopics(topics);
+ var builder = allHaveTopicIds
+ ? TxnOffsetCommitRequest.Builder.forTopicIdsOrNames(data, isTransactionV2Enabled(), true)
Review Comment:
this opens the door for the Unknown_topic_id in the response. Is that coming in a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #22443:
URL: https://github.com/apache/kafka/pull/22443
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3341874276
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition, OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
Review Comment:
agreed, makes sense (I was thinking about case 2, deleted)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3339344581
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition, OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
+ var requestTopicsByName = new HashMap<String, TxnOffsetCommitRequestTopic>();
+ var topicNamesByIds = new HashMap<Uuid, String>();
+ var topics = new ArrayList<TxnOffsetCommitRequestTopic>();
+ var allHaveTopicIds = true;
+ for (var entry : offsets.entrySet()) {
+ var tp = entry.getKey();
+ var offsetAndMetadata = entry.getValue();
+ pendingTxnOffsetCommits.put(
+ tp,
+ new CommittedOffset(
+ offsetAndMetadata.offset(),
+ offsetAndMetadata.metadata(),
+ offsetAndMetadata.leaderEpoch()
+ )
+ );
+ var topicId = topicIds.getOrDefault(tp.topic(), Uuid.ZERO_UUID);
+ allHaveTopicIds &= !topicId.equals(Uuid.ZERO_UUID);
+ var topic = requestTopicsByName.computeIfAbsent(tp.topic(), name -> {
+ var t = new TxnOffsetCommitRequestTopic().setName(name).setTopicId(topicId);
+ topics.add(t);
+ if (!topicId.equals(Uuid.ZERO_UUID)) {
+ topicNamesByIds.put(topicId, name);
+ }
+ return t;
+ });
+ topic.partitions().add(new TxnOffsetCommitRequestData.TxnOffsetCommitRequestPartition()
+ .setPartitionIndex(tp.partition())
+ .setCommittedOffset(offsetAndMetadata.offset())
+ .setCommittedLeaderEpoch(offsetAndMetadata.leaderEpoch().orElse(RecordBatch.NO_PARTITION_LEADER_EPOCH))
+ .setCommittedMetadata(offsetAndMetadata.metadata()));
}
- final TxnOffsetCommitRequestData data = new TxnOffsetCommitRequestData()
+ var data = new TxnOffsetCommitRequestData()
.setTransactionalId(transactionalId)
.setGroupId(groupMetadata.groupId())
.setProducerId(producerIdAndEpoch.producerId)
.setProducerEpoch(producerIdAndEpoch.epoch)
.setMemberId(groupMetadata.memberId())
.setGenerationIdOrMemberEpoch(groupMetadata.generationId())
.setGroupInstanceId(groupMetadata.groupInstanceId().orElse(null))
- .setTopics(TxnOffsetCommitRequest.getTopics(pendingTxnOffsetCommits));
- final TxnOffsetCommitRequest.Builder builder =
- TxnOffsetCommitRequest.Builder.forTopicNames(data, isTransactionV2Enabled());
+ .setTopics(topics);
+ var builder = allHaveTopicIds
+ ? TxnOffsetCommitRequest.Builder.forTopicIdsOrNames(data, isTransactionV2Enabled(), true)
Review Comment:
it is already handled by `error.exception() instanceof RetriableException` as `UNKNOWN_TOPIC_ID` is a `RetriableException` exception. there is also a test covering it: `testTxnOffsetCommitRetriesOnUnknownTopicIdAtV6`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3341871997
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = apiVersions;
+ List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);
Review Comment:
not here I would say, we'll handle it in the other PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3337192221
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java:
##########
@@ -1243,29 +1249,59 @@ private TxnRequestHandler addPartitionsToTransactionHandler() {
private TxnOffsetCommitHandler txnOffsetCommitHandler(TransactionalRequestResult result,
Map<TopicPartition, OffsetAndMetadata> offsets,
ConsumerGroupMetadata groupMetadata) {
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- OffsetAndMetadata offsetAndMetadata = entry.getValue();
- CommittedOffset committedOffset = new CommittedOffset(offsetAndMetadata.offset(),
- offsetAndMetadata.metadata(), offsetAndMetadata.leaderEpoch());
- pendingTxnOffsetCommits.put(entry.getKey(), committedOffset);
+ // Resolve topic ids from the metadata cache at request build time.
+ // KafkaProducer.sendOffsetsToTransaction has already ensured the cache
+ // is fresh for these topics, so this is a non-blocking lookup.
+ var topicIds = metadata.topicIds();
Review Comment:
Thinking about the case of Unknown_topic_ID made me notice, we have the metadata check here only (when creating the `txnOffsetCommitHandler`) , but all the retry logic relies on re-enqueueing the same handler, so this won't be triggered on retries I expect (would re-send the same topic-id if we get unknown_topic_id?).
Is the intention to handle that when processing the Unknown_topic_id? (separate PR?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20444: [10/N] Wire topic IDs through TransactionManager (KIP-1319) [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #22443:
URL: https://github.com/apache/kafka/pull/22443#discussion_r3339340630
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -427,6 +427,19 @@ public KafkaProducer(Properties properties, Serializer<K> keySerializer, Seriali
int deliveryTimeoutMs = configureDeliveryTimeout(config, log);
this.apiVersions = apiVersions;
+ List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config);
Review Comment:
ack. let me know if i need to change anything here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org