You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "OmniaGM (via GitHub)" <gi...@apache.org> on 2025/06/13 14:29:36 UTC
[PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
OmniaGM opened a new pull request, #19964:
URL: https://github.com/apache/kafka/pull/19964
- Metadata doesn't have the full view of topicNames to ids during rebootstrap of client or when topic has been deleted/recreated. The solution is to pass down topic id and stop trying to figure it out later in the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3018530103
I deployed the test again with this change. I think running it for 24 hours should give us reasonable confidence that the problem does not reappear. If we are fairly confident that the fix is still valid, we could merge it and worst case block the first RC on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3037235784
@OmniaGM could you please backport it to 4.1 branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3036595933
> [test_failure.txt](https://github.com/user-attachments/files/21044139/test_failure.txt)
>
>
>
> @chia7712 : Attached is the test failure output. It still happened after the latest fix.
I can't reproduce it locally as well. @chia7712 did you had any luck reproducing the timeout locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160224506
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
+ waitForCondition(() => {
+ val topicId = if (admin.listTopics().names().get().contains(topic)) {
+ topicMetadata(admin, topic).topicId()
+ } else Uuid.ZERO_UUID
+ // don't wait for the physical delete
+ deleteTopicWithAdminRaw(admin, topic)
Review Comment:
Delete topic usually faster at deleting the metadata first however `deleteTopic` without the admin raw waits for delete the hard partitions which might take longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152650081
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,14 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry ->
+ entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name()))
+ ).map(Map.Entry::getValue).findFirst().orElse(null);
Review Comment:
We always had this potentail of batch is null this why I raised the comment here https://github.com/apache/kafka/pull/19964#discussion_r2152281872 that should we have IllegalStateException. I updated this to fail with IllegalStateException instead of leaving it like this
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will relay on topic id only
+ * otherwise the method will relay on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (tpId.topic().isEmpty() && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
Review Comment:
added null check there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2178049890
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -892,7 +901,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
.setTopicData(tpd),
useTransactionV1Version
);
- RequestCompletionHandler callback = response -> handleProduceResponse(response, recordsByPartition, time.milliseconds());
+ RequestCompletionHandler callback = response -> handleProduceResponse(response, recordsByPartition, metadata.topicNames(), time.milliseconds());
Review Comment:
Updated this, if this still not working and the 1st patch is better am thinking of going back to something similiar to first one and maybe have a flag in TopicIdPartition for `ignoreTopicName` that make `equals` and `hashcode` works when topic name not there so when we do `get` on `batches <TopicIdPartitions, ProducerBatch>` would get the expected batch without extra iteration that was originally proposed in first commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3017089038
Will wait for @lucasbru confirmation before merging this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152687163
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
> We always fall back on topic name if topic id not passed down
IIRC, the topic name is ignored in RPC when the version is larger than 13. So, is it possible that the broker receives the produce request without both the topic name and id?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-2994379430
@OmniaGM could you please fix the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152650081
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,14 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry ->
+ entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name()))
+ ).map(Map.Entry::getValue).findFirst().orElse(null);
Review Comment:
We always had this potentail of batch is null this why I raised the comment here https://github.com/apache/kafka/pull/19964#discussion_r2152281872 that should we have IllegalStateException. I have throw IllegalStateException instead of leaving it like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2178044309
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +606,19 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
+
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response will be empty.
+ // For older versions, topic id is zero, and we will find the batch based on the topic name.
+ TopicPartition tp = (!r.topicId().equals(Uuid.ZERO_UUID) && topicNames.containsKey(r.topicId())) ?
+ new TopicPartition(topicNames.get(r.topicId()), p.index()) :
+ new TopicPartition(r.name(), p.index());
+
ProducerBatch batch = batches.get(tp);
+ if (batch == null) {
+ throw new IllegalStateException("batch created for " + tp + " can't be found.");
Review Comment:
Updated the message to include these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160224238
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ TopicIdPartition tpId = new TopicIdPartition(r.topicId(), p.index(), r.name());
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(tpId))
+ .map(Map.Entry::getValue).findFirst().orElse(null);
+
+ if (batch == null) {
+ throw new IllegalStateException("batch created for " + tpId + " can't be found, " +
+ "topic might be recreated after the batch creation.");
Review Comment:
No, I was just thinking that this should be IllegalStateException in generally instead of leaving the NPE from the get in the future for any reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3029093279
> Not sure why the cluster keeps reformatting. However, the default [delivery.timeout.ms](https://kafka.apache.org/documentation/#producerconfigs_delivery.timeout.ms) is 120 secs, which is too long for the test. In our case, it's ok for a producer.send() to fail. We just want to make sure that every callback is called. So, we could at least reduce [delivery.timeout.ms](https://kafka.apache.org/documentation/#producerconfigs_delivery.timeout.ms).
The root cause is that the default partition count is '2', while the re-created partition count is '1'. Hence, the produce request sent to topic-1, which the topic is auto-created, can never be completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3036876302
@OmniaGM thanks for handling that crisis. BTW, we have a commit tool https://github.com/apache/kafka/blob/trunk/committer-tools/reviewers.py which can easily add `reviewers` to the commit message. Perhaps you could give it a try next time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152625330
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,14 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry ->
+ entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name()))
Review Comment:
moved the initalisation out of stream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152281872
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,22 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ ProducerBatch batch = null;
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ List<ProducerBatch> matchedBatchesForTopicId = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name())))
+ .map(Map.Entry::getValue)
+ .collect(Collectors.toList());
+
+ if (matchedBatchesForTopicId.size() > 1) {
+ matchedBatchesForTopicId.forEach(matchedBatch ->
+ failBatch(matchedBatch, new RuntimeException("More than one batch with same topic id and partition."), false));
Review Comment:
Removed this now in favour of using first find. I am just thinking should we in general return IllegalStateException if batch is null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152691409
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
It shouldn't happened but if it did the batch would fail with "unkonw topic"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2154081661
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
It's true that `Sender::topicIdsForBatches` could return zero topicId in rare cases. It seems that we could just pass along the mapping to zero topicId when handling the produce response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160223692
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
Review Comment:
made it private as moving the test to the new java test
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
Review Comment:
This was left from the recreation logic of the topic I just handled this in the recreate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3033126276
> I ran the new test until failure. After 10 minutes, I still saw the timeout.
@junrao Could you share more logs with me if they exist. I'm curious about the root cause of the timeout, since it does not happen on my machine even after over 1000 attempts for the https://github.com/apache/kafka/commit/6419e079e439f312de5b2ea4c33def457ef4f8fb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3027149079
I am soaking commit `a5644e92f72e8fd8`. In the first 12 hours, the problem did not reappear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-2979466809
@OmniaGM @junrao I have been running the soak for 24h, and it's looking good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-2980999168
I have added some test in ProducerSendWhileDeletionTest to cover recreation while producing as well hope this will be enough to cover these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152280065
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,22 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ ProducerBatch batch = null;
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ List<ProducerBatch> matchedBatchesForTopicId = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name())))
Review Comment:
Updated this to get first match. I was just over cautious but I don't think I need to be
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-2975852521
@junrao I redeployed the soak with the fix. I will report back if the problem reoccurs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3036879000
> @OmniaGM thanks for handling that crisis. BTW, we have a commit tool https://github.com/apache/kafka/blob/trunk/committer-tools/reviewers.py which can easily add `reviewers` to the commit message. Perhaps you could give it a try next time.
I was wondering about this just now as I noticed the commit went without them. I thought it runs automatically. My bad will try it next time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-2998022542
Also, regarding the title of the PR. "Not relaying on metadata to map between topic id and name".
We are still relying on the metadata to map topic id and name. We just want to use consistent metadata between generating the produce request and handing the produce response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2161994977
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will rely on topic id only
+ * otherwise the method will rely on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (Utils.isBlank(tpId.topic()) && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return topicPartition.equals(tpId.topicPartition());
Review Comment:
> In the rare case that Sender::topicIdsForBatches returns 0 topic id (e.g. topic is deleted), we will pass along topicName -> 0 to handleProduceResponse(). The response will include empty topic and 0 topic id. It's important that we find a match in this case to avoid IllegalStateException. I am thinking that we should first try to do the comparison on topic name, if it's not empty. Otherwise, just do the comparison on topic id even if it's zero.
Thinking out loud here shouldn't this be true for `TopicIdPartition::equals` as we might have this rare situation in other places in the code where topic id partition metadata wouldn't exist fully? Specially that right now not all Kafka a topic id aware yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160230646
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will rely on topic id only
+ * otherwise the method will rely on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (Utils.isBlank(tpId.topic()) && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return topicPartition.equals(tpId.topicPartition());
Review Comment:
What about change `ProducerBatch` to use `TopicIdPartition` instead of `TopicPartition` this will simplify things instead of all of these checks and metadata lookups also I wouldn' tneed to filter batches to find the topicId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152693786
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
The sender on client side always fetch metadata before send so we should get some sort of metadata to populate topic id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2153260298
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,22 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will relay on topic id only
+ * otherwise the method will relay on topic name.
Review Comment:
Nit: I believe this is meant to be _rely_:
```suggestion
* If topic name is empty and topic id is persisted then the method will rely on topic id only
* otherwise the method will rely on topic name.
```
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,22 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will relay on topic id only
+ * otherwise the method will relay on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ boolean emptyTopicName = tpId.topic() == null || tpId.topic().isEmpty();
Review Comment:
Minor suggestion—`Utils.isBlank()` is a little more succinct and catches cases where the string is whitespace only.
```suggestion
boolean emptyTopicName = Utils.isBlank(tpId.topic());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2180133482
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -892,7 +902,8 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
.setTopicData(tpd),
useTransactionV1Version
);
- RequestCompletionHandler callback = response -> handleProduceResponse(response, recordsByPartition, time.milliseconds());
+ Map<Uuid, String> topicNames = metadata.topicNames();
Review Comment:
it would be better to add comments :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2178023461
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -892,7 +901,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
.setTopicData(tpd),
useTransactionV1Version
);
- RequestCompletionHandler callback = response -> handleProduceResponse(response, recordsByPartition, time.milliseconds());
+ RequestCompletionHandler callback = response -> handleProduceResponse(response, recordsByPartition, metadata.topicNames(), time.milliseconds());
Review Comment:
@lucasbru : Thanks for reporting the issue. I think this is the problem. This callback is called when the response completes, at which point, the metadata could have changed. Calling `metadata.topicNames()` at that point will potentially return metadata inconsistent with what's being used when generating the produce request. So, we need to call `metadata.topicNames()` outside the callback, save it as a final and pass it to the callback.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +606,19 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
+
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response will be empty.
+ // For older versions, topic id is zero, and we will find the batch based on the topic name.
+ TopicPartition tp = (!r.topicId().equals(Uuid.ZERO_UUID) && topicNames.containsKey(r.topicId())) ?
+ new TopicPartition(topicNames.get(r.topicId()), p.index()) :
+ new TopicPartition(r.name(), p.index());
+
ProducerBatch batch = batches.get(tp);
+ if (batch == null) {
+ throw new IllegalStateException("batch created for " + tp + " can't be found.");
Review Comment:
It would be useful to include `topicNames`, `r.topicId()` and `r.name()` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3032990255
> I made the following change and the test hasn't failed after running for 15 minutes. It uses a shorter delivery timeout and request timeout. We probably also want to reduce the test time.
>
> ```
> AtomicInteger numAcks = new AtomicInteger(0);
> var producerFuture = CompletableFuture.runAsync(() -> {
> - try (var producer = cluster.producer()) {
> + try (var producer = createProducer()) {
> ```
Updated the pr now, and removed the timeouts seems to be running way faster now and should be within the default 60 timeout for the test (at least locally will wait for the pipeline)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152340372
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,14 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry ->
+ entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name()))
+ ).map(Map.Entry::getValue).findFirst().orElse(null);
Review Comment:
It is possible to have null batch, right? For example, the topic is recreated after the batch is generated
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,14 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry ->
+ entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name()))
Review Comment:
Should we create the `TopicIdPartition` outside the stream to avoid creating many temporary objects?
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will relay on topic id only
+ * otherwise the method will relay on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (tpId.topic().isEmpty() && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
Review Comment:
`TopicIdPartition#topic` is nullable, so the null check is necessary.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
one question: what happens if the topic id is unknown (`Uuid.ZERO_UUID`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2161994977
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will rely on topic id only
+ * otherwise the method will rely on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (Utils.isBlank(tpId.topic()) && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return topicPartition.equals(tpId.topicPartition());
Review Comment:
> In the rare case that Sender::topicIdsForBatches returns 0 topic id (e.g. topic is deleted), we will pass along topicName -> 0 to handleProduceResponse(). The response will include empty topic and 0 topic id. It's important that we find a match in this case to avoid IllegalStateException. I am thinking that we should first try to do the comparison on topic name, if it's not empty. Otherwise, just do the comparison on topic id even if it's zero.
Thinking out loud here shouldn't this be true for `TopicIdPartition::equals` as we might have this rare situation in other places in the code where topic partition metadata wouldn't exist fully? Specially that right now not all Kafka a topic id aware yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160224506
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
+ waitForCondition(() => {
+ val topicId = if (admin.listTopics().names().get().contains(topic)) {
+ topicMetadata(admin, topic).topicId()
+ } else Uuid.ZERO_UUID
+ // don't wait for the physical delete
+ deleteTopicWithAdminRaw(admin, topic)
Review Comment:
Delete topic usually faster as it deletes the metadata first however `deleteTopic` without the admin raw waits for delete the hard partitions which might take longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3023066199
Unfortunately, this version again seems to be triggering a problem with producers getting stuck during flushing. We are running with Kafka commitId `405d1b7ed8b6c452` which should be the latest version of this PR.
In the thread dump, I see producers being stuck here:
```
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)
- parking to wait for <0x000000072a34ed28> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.12/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@17.0.12/AbstractQueuedSynchronizer.java:1047)
at java.util.concurrent.CountDownLatch.await(java.base@17.0.12/CountDownLatch.java:230)
at org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:76)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:1075)
at org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:1325)
at ...
```
The last sign of life from the producer is this:
```
soak_44.244.234.161/streams.log.2025-06-30-22:59:[2025-06-30 22:50:32,044] ERROR [kafka-producer-network-thread | i-0a00080f294c267cf-StreamThread-3-producer] [Producer clientId=i-0a00080f294c267cf-StreamThread-3-producer] Uncaught error in request completion: (org.apache.kafka.clients.NetworkClient)
soak_44.244.234.161/streams.log.2025-06-30-22:59-java.lang.IllegalStateException: batch created for -1 can't be found.
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:620)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$3(Sender.java:597)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.lang.Iterable.forEach(Iterable.java:75)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:597)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$9(Sender.java:904)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:669)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:661)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:340)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242)
soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.lang.Thread.run(Thread.java:840)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3032800027
@OmniaGM : Thanks for the updated PR. I ran the new test until failure. After 10 minutes, I still saw the timeout.
```
java.util.concurrent.TimeoutException: testSendWhileTopicGetRecreated() timed out after 120 seconds
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6089077873146292020/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6089077873146292020/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6089077873146292020/controller_3000 with metadata.version 4.2-IV1.
[2025-07-03 08:32:04,927] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:05,103] WARN [Producer clientId=producer-2] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:05,214] WARN [Producer clientId=producer-2] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:05,438] WARN [Producer clientId=producer-2] The metadata response from the cluster reported a recoverable issue with correlation id 9 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:06,165] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6608140775505501629/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6608140775505501629/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6608140775505501629/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:06,290] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:06,450] WARN [Producer clientId=producer-3] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:06,558] WARN [Producer clientId=producer-3] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:07,360] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1689995402898229137/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1689995402898229137/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1689995402898229137/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:07,468] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:07,628] WARN [Producer clientId=producer-4] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:07,733] WARN [Producer clientId=producer-4] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:07,956] WARN [Producer clientId=producer-4] The metadata response from the cluster reported a recoverable issue with correlation id 8 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:08,770] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-17048505300526058547/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-17048505300526058547/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-17048505300526058547/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:08,880] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:09,145] WARN [Producer clientId=producer-5] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:09,272] WARN [Producer clientId=producer-5] Got error produce response with correlation id 8 on topic-partition topic-0, retrying (2147483646 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender:680)
[2025-07-03 08:32:09,273] WARN [Producer clientId=producer-5] Received unknown topic or partition error in produce request on partition topic-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender:702)
[2025-07-03 08:32:09,365] WARN [Producer clientId=producer-5] The metadata response from the cluster reported a recoverable issue with correlation id 9 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:09,484] WARN [Producer clientId=producer-5] The metadata response from the cluster reported a recoverable issue with correlation id 10 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:10,164] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-12586289379138957056/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-12586289379138957056/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-12586289379138957056/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:10,281] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:10,440] WARN [Producer clientId=producer-6] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:10,545] WARN [Producer clientId=producer-6] The metadata response from the cluster reported a recoverable issue with correlation id 8 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:10,782] WARN [Producer clientId=producer-6] The metadata response from the cluster reported a recoverable issue with correlation id 9 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:11,569] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-2477307315647262884/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-2477307315647262884/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-2477307315647262884/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:11,712] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:11,857] WARN [Producer clientId=producer-7] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:12,066] WARN [Producer clientId=producer-7] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:12,760] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14757478012604431632/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14757478012604431632/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14757478012604431632/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:12,860] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:13,007] WARN [Producer clientId=producer-8] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:13,217] WARN [Producer clientId=producer-8] Got error produce response with correlation id 9 on topic-partition topic-1, retrying (2147483646 attempts left). Error: UNKNOWN_TOPIC_ID (org.apache.kafka.clients.producer.internals.Sender:680)
[2025-07-03 08:32:13,217] WARN [Producer clientId=producer-8] Received invalid metadata error in produce request on partition topic-1 due to org.apache.kafka.common.errors.UnknownTopicIdException: This server does not host this topic ID. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender:706)
[2025-07-03 08:32:13,331] WARN [Producer clientId=producer-8] Got error produce response with correlation id 12 on topic-partition topic-1, retrying (2147483645 attempts left). Error: UNKNOWN_TOPIC_ID (org.apache.kafka.clients.producer.internals.Sender:680)
[2025-07-03 08:32:13,331] WARN [Producer clientId=producer-8] Received invalid metadata error in produce request on partition topic-1 due to org.apache.kafka.common.errors.UnknownTopicIdException: This server does not host this topic ID. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender:706)
[2025-07-03 08:32:13,332] WARN [Producer clientId=producer-8] The metadata response from the cluster reported a recoverable issue with correlation id 13 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:13,551] WARN [Producer clientId=producer-8] Got error produce response with correlation id 15 on topic-partition topic-1, retrying (2147483644 attempts left). Error: UNKNOWN_TOPIC_ID (org.apache.kafka.clients.producer.internals.Sender:680)
[2025-07-03 08:32:13,551] WARN [Producer clientId=producer-8] Received invalid metadata error in produce request on partition topic-1 due to org.apache.kafka.common.errors.UnknownTopicIdException: This server does not host this topic ID. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender:706)
[2025-07-03 08:32:14,376] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-321361512033974143/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-321361512033974143/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-321361512033974143/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:14,462] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:14,620] WARN [Producer clientId=producer-9] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:14,726] WARN [Producer clientId=producer-9] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:14,965] WARN [Producer clientId=producer-9] The metadata response from the cluster reported a recoverable issue with correlation id 9 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:15,774] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-543793708133122389/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-543793708133122389/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-543793708133122389/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:15,883] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:16,033] WARN [Producer clientId=producer-10] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:16,241] WARN [Producer clientId=producer-10] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:16,472] WARN [Producer clientId=producer-10] The metadata response from the cluster reported a recoverable issue with correlation id 9 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:17,360] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-18238593010590373133/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-18238593010590373133/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-18238593010590373133/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:17,454] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:17,634] WARN [Producer clientId=producer-11] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:17,739] WARN [Producer clientId=producer-11] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:18,369] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1007440053127284728/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1007440053127284728/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-1007440053127284728/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-03 08:32:18,459] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-03 08:32:18,636] WARN [Producer clientId=producer-12] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:18,845] WARN [Producer clientId=producer-12] The metadata response from the cluster reported a recoverable issue with correlation id 8 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:19,035] WARN [Producer clientId=producer-12] The metadata response from the cluster reported a recoverable issue with correlation id 10 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-03 08:32:19,997] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-11984181535268137544/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-11984181535268137544/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-11984181535268137544/broker_1_data0 with metadata.version 4.2-IV1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3028419156
@OmniaGM Still looking good - problem seems to be resovled, but the integration test doesn't seem to pass now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3028806833
@lucasbru : Thanks for the update. As for the integration test failure. I saw the following locally.
```
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-13068287464958241505/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-13068287464958241505/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-13068287464958241505/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-02 09:49:51,304] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-02 09:49:51,469] WARN [Producer clientId=producer-2] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:51,677] WARN [Producer clientId=producer-2] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:52,372] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14231399981533981299/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14231399981533981299/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-14231399981533981299/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-02 09:49:52,491] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-02 09:49:52,648] WARN [Producer clientId=producer-3] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:52,858] WARN [Producer clientId=producer-3] The metadata response from the cluster reported a recoverable issue with correlation id 8 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:53,568] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6584854305397606511/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6584854305397606511/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-6584854305397606511/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-02 09:49:53,671] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-02 09:49:53,820] WARN [Producer clientId=producer-4] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:54,566] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-7033202108695639212/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-7033202108695639212/broker_0_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-7033202108695639212/broker_1_data0 with metadata.version 4.2-IV1.
[2025-07-02 09:49:54,652] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-02 09:49:54,811] WARN [Producer clientId=producer-5] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:55,023] WARN [Producer clientId=producer-5] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:55,590] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-4645572128043181693/controller_3000 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-4645572128043181693/broker_1_data0 with metadata.version 4.2-IV1.
Formatting metadata directory /var/folders/t1/_lmg790n4dg46yk7lzvzg45r0000gp/T/kafka-4645572128043181693/broker_0_data0 with metadata.version 4.2-IV1.
[2025-07-02 09:49:55,693] WARN [QuorumController id=3000] Performing controller activation. The metadata log appears to be empty. Appending 6 bootstrap record(s) in metadata transaction at metadata.version 4.2-IV1 from bootstrap source 'testkit'. (org.apache.kafka.controller.QuorumController:106)
[2025-07-02 09:49:55,847] WARN [Producer clientId=producer-6] The metadata response from the cluster reported a recoverable issue with correlation id 1 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:56,058] WARN [Producer clientId=producer-6] The metadata response from the cluster reported a recoverable issue with correlation id 7 : {topic=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1295)
[2025-07-02 09:49:56,568] WARN [NodeToControllerChannelManager id=3000 name=registration] Attempting to close NetworkClient that has already been closed. (org.apache.kafka.clients.NetworkClient:744)
```
Not sure why the cluster keeps reformatting. However, the default [delivery.timeout.ms](https://kafka.apache.org/documentation/#producerconfigs_delivery.timeout.ms) is 120 secs, which is too long for the test. In our case, it's ok for a producer.send() to fail. We just want to make sure that every callback is called. So, we could at least reduce [delivery.timeout.ms](https://kafka.apache.org/documentation/#producerconfigs_delivery.timeout.ms).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3033176941
[test_failure.txt](https://github.com/user-attachments/files/21044139/test_failure.txt)
@chia7712 : Attached is the test failure output. It still happened after the latest fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2146568617
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,22 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ ProducerBatch batch = null;
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ List<ProducerBatch> matchedBatchesForTopicId = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name())))
+ .map(Map.Entry::getValue)
+ .collect(Collectors.toList());
+
+ if (matchedBatchesForTopicId.size() > 1) {
+ matchedBatchesForTopicId.forEach(matchedBatch ->
+ failBatch(matchedBatch, new RuntimeException("More than one batch with same topic id and partition."), false));
Review Comment:
If this is unexpected, we want to throw IllegalStateException as in other places in this file
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,18 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (tpId.topic().isEmpty()) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return this.equals(tpId);
Review Comment:
If topic name is not empty, it means that the topicId is zero. In this case, we just want to match based on topic name, right?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,22 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ ProducerBatch batch = null;
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ List<ProducerBatch> matchedBatchesForTopicId = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(new TopicIdPartition(r.topicId(), p.index(), r.name())))
Review Comment:
Could we just find the first match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152276755
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +78,18 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (tpId.topic().isEmpty()) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return this.equals(tpId);
Review Comment:
Updated this now to be sepcific about topic name exist but id doesnt or other way around
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3024420820
> Unfortunately, this version again seems to be triggering a problem with producers getting stuck during flushing. We are running with Kafka commitId `405d1b7ed8b6c452` which should be the latest version of this PR.
>
> In the thread dump, I see producers being stuck here:
>
> ```
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)
> - parking to wait for <0x000000072a34ed28> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:211)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.12/AbstractQueuedSynchronizer.java:715)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@17.0.12/AbstractQueuedSynchronizer.java:1047)
> at java.util.concurrent.CountDownLatch.await(java.base@17.0.12/CountDownLatch.java:230)
> at org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:76)
> at org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:1075)
> at org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:1325)
> at ...
> ```
>
> The last sign of life from the producer is this:
>
> ```
> soak_44.244.234.161/streams.log.2025-06-30-22:59:[2025-06-30 22:50:32,044] ERROR [kafka-producer-network-thread | i-0a00080f294c267cf-StreamThread-3-producer] [Producer clientId=i-0a00080f294c267cf-StreamThread-3-producer] Uncaught error in request completion: (org.apache.kafka.clients.NetworkClient)
> soak_44.244.234.161/streams.log.2025-06-30-22:59-java.lang.IllegalStateException: batch created for -1 can't be found.
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:620)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$3(Sender.java:597)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.lang.Iterable.forEach(Iterable.java:75)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:597)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$9(Sender.java:904)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:669)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:661)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:340)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242)
> soak_44.244.234.161/streams.log.2025-06-30-22:59- at java.base/java.lang.Thread.run(Thread.java:840)
> ```
This is basically same state as NPE but with `IllegalStateException` where tooic name is empty and topic id not part of initial metadata passed down from `sendProduceRequest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2178056683
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +606,20 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
+
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response will be empty.
+ // For older versions, topic id is zero, and we will find the batch based on the topic name.
+ TopicPartition tp = (!r.topicId().equals(Uuid.ZERO_UUID) && topicNames.containsKey(r.topicId())) ?
+ new TopicPartition(topicNames.get(r.topicId()), p.index()) :
+ new TopicPartition(r.name(), p.index());
+
ProducerBatch batch = batches.get(tp);
+ if (batch == null) {
+ throw new IllegalStateException("Can't find batch created for topic id " + r.topicId() +
+ " topic name " + r.name() + "-" + p.index() + " using " + topicNames);
Review Comment:
"-" + p.index() => " partition " + p.index() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3024789540
@lucasbru can you give this another go please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3032861053
I made the following change and the test hasn't failed after running for 15 minutes. It uses a shorter delivery timeout and request timeout. We probably also want to reduce the test time.
```
AtomicInteger numAcks = new AtomicInteger(0);
var producerFuture = CompletableFuture.runAsync(() -> {
- try (var producer = cluster.producer()) {
+ try (var producer = createProducer()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM merged PR #19964:
URL: https://github.com/apache/kafka/pull/19964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152668819
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
We always fall back on topic name if topic id not passed down in the produce request and sort it out down the line when the broker recieve the produce request. It should be the same as if the client not topic id aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2152719515
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -855,7 +860,7 @@ private void sendProduceRequest(long now, int destination, short acks, int timeo
if (batches.isEmpty())
return;
- final Map<TopicPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
+ final Map<TopicIdPartition, ProducerBatch> recordsByPartition = new HashMap<>(batches.size());
Map<String, Uuid> topicIds = topicIdsForBatches(batches);
Review Comment:
Actually there is a potential for this in `Sender::topicIdsForBatches` default to Uuid zero if metadata doesn't have the topic. Am thinking now that maybe `ProducerBatch` should use `TopicIdPartition` instead of `TopicPartition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160223873
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
Review Comment:
I changed the test a bit here now there is a counter for success records that get updated in the callback and I verify this at the end of the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2160224506
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
+ waitForCondition(() => {
+ val topicId = if (admin.listTopics().names().get().contains(topic)) {
+ topicMetadata(admin, topic).topicId()
+ } else Uuid.ZERO_UUID
+ // don't wait for the physical delete
+ deleteTopicWithAdminRaw(admin, topic)
Review Comment:
Delete topic usually faster at deleting the metadata first however `deleteTopic` without the admin raw waits for deleting the hard partitions which might take longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2156322849
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
Review Comment:
same this object => this object
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
Review Comment:
Since the topic could be deleted, it's possible for the metadata to have -1, right?
Also, could we verify that every record's callback is called?
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
Review Comment:
Hmm, where would a TopicExistsException be thrown?
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
+ waitForCondition(() => {
+ val topicId = if (admin.listTopics().names().get().contains(topic)) {
+ topicMetadata(admin, topic).topicId()
+ } else Uuid.ZERO_UUID
+ // don't wait for the physical delete
+ deleteTopicWithAdminRaw(admin, topic)
Review Comment:
Hmm, `deleteTopicWithAdminRaw()` doesn't wait for the metadata propagation to the brokers. However, the producer only sees the deleted topic after the metadata is propagated. Is this test effective?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ TopicIdPartition tpId = new TopicIdPartition(r.topicId(), p.index(), r.name());
+ ProducerBatch batch = batches.entrySet().stream()
Review Comment:
This changes a map lookup to an iteration. Could we do some produce perf test (with multiple topic/partitions) to verify there is no performance degradation?
##########
core/src/test/scala/integration/kafka/api/ProducerSendWhileDeletionTest.scala:
##########
@@ -81,6 +86,59 @@ class ProducerSendWhileDeletionTest extends IntegrationTestHarness {
assertEquals(topic, producer.send(new ProducerRecord(topic, null, "value".getBytes(StandardCharsets.UTF_8))).get.topic())
}
+ @Test
+ def testSendWhileTopicGetRecreated(): Unit = {
+ val numRecords = 10
+ val topic = "topic"
+ val admin = createAdminClient()
+ val producer = createProducer()
+
+ try {
+ val fs = CompletableFuture.runAsync(() => {
+ for (_ <- 1 to 20) {
+ recreateTopic(admin, topic)
+ }
+ })
+ val producerFutures = new util.ArrayList[CompletableFuture[Void]]
+ for (_ <- 0 until numRecords) {
+ producerFutures.add(CompletableFuture.runAsync(() => {
+ for (i <- 0 until numRecords) {
+ producer.send(new ProducerRecord(topic, null, ("value" + i).getBytes(StandardCharsets.UTF_8)), new Callback() {
+ override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = {
+ assertNotNull(metadata)
+ assertNotEquals(metadata.offset, -1L)
+ assertNotEquals(metadata.timestamp, RecordBatch.NO_TIMESTAMP)
+ }
+ }).get()
+ }
+ }))
+ }
+ fs.join()
+ producerFutures.forEach(_.join)
+ } catch {
+ case e: TopicExistsException =>
+ admin.deleteTopics(util.List.of(topic)).all().get()
+ } finally {
+ admin.close()
+ producer.close()
+ }
+ }
+
+ def recreateTopic(admin: Admin, topic: String): Unit = {
Review Comment:
Could this be private?
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will rely on topic id only
+ * otherwise the method will rely on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (Utils.isBlank(tpId.topic()) && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return topicPartition.equals(tpId.topicPartition());
Review Comment:
In the rare case that `Sender::topicIdsForBatches` returns 0 topic id (e.g. topic is deleted), we will pass along topicName -> 0 to handleProduceResponse(). The response will include empty topic and 0 topic id. It's important that we find a match in this case to avoid IllegalStateException. I am thinking that we should first try to do the comparison on topic name, if it's not empty. Otherwise, just do the comparison on topic id even if it's zero.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ TopicIdPartition tpId = new TopicIdPartition(r.topicId(), p.index(), r.name());
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(tpId))
+ .map(Map.Entry::getValue).findFirst().orElse(null);
+
+ if (batch == null) {
+ throw new IllegalStateException("batch created for " + tpId + " can't be found, " +
+ "topic might be recreated after the batch creation.");
Review Comment:
Hmm, the recreation of the topic shouldn't hit this IllegalStateException, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Not relaying on metadata to map between topic id and name. [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19964:
URL: https://github.com/apache/kafka/pull/19964#discussion_r2162542690
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
Review Comment:
How about the following?
// Version 13 drop topic name and add support to topic id.
// We need to find batch based on topic id and partition index only as
// topic name in the response will be empty.
// For older versions, topic id is zero and we will find the batch based on the topic name.
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
+ * If topic name is empty and topic id is persisted then the method will rely on topic id only
+ * otherwise the method will rely on topic name.
+ * @return true if topic has same topicId and partition index as topic names some time might be empty.
+ */
+ public boolean same(TopicIdPartition tpId) {
+ if (Utils.isBlank(tpId.topic()) && !tpId.topicId.equals(Uuid.ZERO_UUID)) {
+ return topicId.equals(tpId.topicId) &&
+ topicPartition.partition() == tpId.partition();
+ } else {
+ return topicPartition.equals(tpId.topicPartition());
Review Comment:
Actually, what I said about topic deletion wasn't correct. `sendProduceRequest()` is called in the Sender thread and at that point, the metadata cached in the client won't change. We drain `ProducerBatch` based on the topic/partition in the metadata. The `topicIds` map created is based on the same metadata. So, if a partition is included in a produce request, the `topicIds` should always include that topic (assuming the topic Id is supported on the server side), even though the server may have deleted that topic. So, this code is fine.
Regarding whether to use `TopicIdPartition` in `ProducerBatch`. In the rare case, topicId could change over time for the same topic. So, we probably can't store `TopicIdPartition` as a final field when `ProducerBatch` is created. The easiest way to do this is probably to bind the topicId when the ProducerBatch is drained, based on the metadata at that time. This is more or less what this PR does.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/producer/ProducerSendWhileDeletionTest.java:
##########
@@ -157,6 +164,42 @@ public void testSendWithRecreatedTopic() throws Exception {
}
}
+ @Timeout(90)
+ @ClusterTest
+ public void testSendWhileTopicGetRecreated() {
+ int maxNumRecreatTopicAttempts = 10;
+ List<Uuid> topicIds = new CopyOnWriteArrayList<>();
+ var recreateTopicFuture = CompletableFuture.runAsync(() -> {
+ for (int i = 1; i <= maxNumRecreatTopicAttempts; i++) {
+ Uuid topicId = recreateTopic();
+ if (topicId != Uuid.ZERO_UUID) {
+ topicIds.add(topicId);
+ }
+ }
+ });
+
+ AtomicInteger numSuccess = new AtomicInteger(0);
+ var producerFutures = IntStream.range(0, 2).mapToObj(producerIndex -> CompletableFuture.runAsync(() -> {
+ try (var producer = cluster.producer()) {
+ for (int i = 1; i <= numRecords; i++) {
+ var resp = producer.send(new ProducerRecord<>(topic, null, ("value" + i).getBytes()),
+ (metadata, exception) -> {
+ if (metadata != null) {
+ numSuccess.incrementAndGet();
+ }
+ }).get();
+ assertEquals(resp.topic(), topic);
+ }
+ } catch (Exception e) {
+ // ignore
+ }
+ })).toList();
+ recreateTopicFuture.join();
+ producerFutures.forEach(CompletableFuture::join);
+ assertTrue(Math.abs(maxNumRecreatTopicAttempts - topicIds.size()) <= 5);
+ assertEquals(20, numSuccess.intValue());
Review Comment:
20 => 2 * numRecords ?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ TopicIdPartition tpId = new TopicIdPartition(r.topicId(), p.index(), r.name());
+ ProducerBatch batch = batches.entrySet().stream()
Review Comment:
An alternative is for `handleProduceResponse()` to take a Map<TopicPartition, ProducerBatch> and a Map<UUID, String>. If the response has non-zero topicId, we look up the second map to find the topic name and then use the first map to find the batch. Otherwise, we look up the first map using the topic name.
##########
clients/src/main/java/org/apache/kafka/common/TopicIdPartition.java:
##########
@@ -78,6 +80,21 @@ public TopicPartition topicPartition() {
return topicPartition;
}
+ /**
+ * Checking if TopicIdPartition meant to be the same reference to same this object but doesn't have all the data.
Review Comment:
Hmm, it seems this isn't fixed?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/producer/ProducerSendWhileDeletionTest.java:
##########
@@ -157,6 +164,42 @@ public void testSendWithRecreatedTopic() throws Exception {
}
}
+ @Timeout(90)
+ @ClusterTest
+ public void testSendWhileTopicGetRecreated() {
+ int maxNumRecreatTopicAttempts = 10;
+ List<Uuid> topicIds = new CopyOnWriteArrayList<>();
+ var recreateTopicFuture = CompletableFuture.runAsync(() -> {
+ for (int i = 1; i <= maxNumRecreatTopicAttempts; i++) {
+ Uuid topicId = recreateTopic();
+ if (topicId != Uuid.ZERO_UUID) {
+ topicIds.add(topicId);
+ }
+ }
+ });
+
+ AtomicInteger numSuccess = new AtomicInteger(0);
+ var producerFutures = IntStream.range(0, 2).mapToObj(producerIndex -> CompletableFuture.runAsync(() -> {
+ try (var producer = cluster.producer()) {
+ for (int i = 1; i <= numRecords; i++) {
+ var resp = producer.send(new ProducerRecord<>(topic, null, ("value" + i).getBytes()),
+ (metadata, exception) -> {
+ if (metadata != null) {
+ numSuccess.incrementAndGet();
+ }
+ }).get();
+ assertEquals(resp.topic(), topic);
+ }
+ } catch (Exception e) {
+ // ignore
+ }
+ })).toList();
+ recreateTopicFuture.join();
+ producerFutures.forEach(CompletableFuture::join);
+ assertTrue(Math.abs(maxNumRecreatTopicAttempts - topicIds.size()) <= 5);
Review Comment:
Hmm, why doesn't each topic creation create a new topic Id?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##########
@@ -609,7 +607,18 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
.collect(Collectors.toList()),
p.errorMessage(),
p.currentLeader());
- ProducerBatch batch = batches.get(tp);
+ // Version 13 drop topic name and add support to topic id.
+ // We need to find batch based on topic id and partition index only as
+ // topic name in the response might be empty.
+ TopicIdPartition tpId = new TopicIdPartition(r.topicId(), p.index(), r.name());
+ ProducerBatch batch = batches.entrySet().stream()
+ .filter(entry -> entry.getKey().same(tpId))
+ .map(Map.Entry::getValue).findFirst().orElse(null);
+
+ if (batch == null) {
+ throw new IllegalStateException("batch created for " + tpId + " can't be found, " +
+ "topic might be recreated after the batch creation.");
Review Comment:
It's find to throw IllegalStateException here. We just need to adjust the error message since recreating a topic shouldn't cause this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19397: Ensure consistent metadata usage in produce request and response [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #19964:
URL: https://github.com/apache/kafka/pull/19964#issuecomment-3036856692
I ran an existing test `testSendWithTopicReassignmentIsMidWay` until failure and hit the same failure
([test_failure2.txt](https://github.com/user-attachments/files/21062667/test_failure2.txt)). So, this seems to be issue in the test framework on my local environment. We can merge the PR as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org