You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "shashankhs11 (via GitHub)" <gi...@apache.org> on 2025/07/29 01:05:49 UTC
[PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
shashankhs11 opened a new pull request, #20254:
URL: https://github.com/apache/kafka/pull/20254
Created a new integration test that fails indicating that the issue is indeed a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3363717246
@shashankhs11 : Thanks. Since the fix doesn't change any public API, no KIP is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3363642966
> Would you be willing to submit a patch?
@junrao Yes, I would love to. I will look into this and try to get a better understanding of what is required.
One clarifying question - Would we need a KIP for this? Or can a patch be submitted in the same PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3367507028
@junrao patch is available for review.
This change seemed to have fixed the infinite retry behavior but the offsets were still being committed and I am not sure what could be the reason for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: Add test to verify data-loss bug was fixed inside the producer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #20254:
URL: https://github.com/apache/kafka/pull/20254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3153158682
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3264447960
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3369787851
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3138158729
> Since, you've confirmed that this is indeed a bug, I would like to start by writing failing unit tests that are more targeted and commit within the same PR
Given that the issue is in the producer, it might be simpler/better to just try writing a more tailored unit test in a new PR? Ie, so couple of `sends()` which are supported to create a batch that is too large, and call `producer.flush()` which would not return but eventually fails with a `delivery.timeout.ms` error (this error might be returned in the `producer.send(..., new Callback())`.
Or you try to go even one level deeper, and try to unit test `org.apache.kafka.clients.producer.internals.Sender` in combination with the passed in `RecordAccumulator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3152755901
@mjsax : Hmm, the splitting code has some logic to chain the response futures. When we append a record to a new batch after a split, we create a new future for the record and chain it to the old future.
```
private boolean tryAppendForSplit(long timestamp, ByteBuffer key, ByteBuffer value, Header[] headers, Thunk thunk) {
...
FutureRecordMetadata future = new FutureRecordMetadata(this.produceFuture, this.recordCount,
timestamp,
key == null ? -1 : key.remaining(),
value == null ? -1 : value.remaining(),
Time.SYSTEM);
// Chain the future to the original thunk.
thunk.future.chain(future);
```
`FutureRecordMetadata.chain()` then adds the new future as a dependency.
```
void chain(FutureRecordMetadata futureRecordMetadata) {
if (nextRecordMetadata == null)
nextRecordMetadata = futureRecordMetadata;
else
nextRecordMetadata.chain(futureRecordMetadata);
}
```
`FutureRecordMetadata.get()` will then wait until all chained response futures complete.
```
public RecordMetadata get() throws InterruptedException, ExecutionException {
this.result.await();
if (nextRecordMetadata != null)
return nextRecordMetadata.get();
return valueOrError();
}
```
So, a pending `flush()` waiting for an original batch is not supposed to complete until all split batches have completed. Did you actually observe that `flush()` returns early?
@lianetm : Currently, `KafkaProducer.doSend()` only rejects a record if its size is > max.request.size. It's ok if a record has a size larger than batch.size. We will just create a larger new batch to accommodate for it. So, during split, it's reasonable to follow the same approach to allow the first record in a new batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3134343060
Thank you so much for the detailed explanation @mjsax. I have not yet started my investigation into this, but i think this really helps me and gives me a solid starting point to trace the bug. Since, you've confirmed that this is indeed a bug, I would like to start by writing failing unit tests that are more targeted and commit within the same PR. I think this would make it easier for everyone to review and propose a fix/approach since your initial impression was that it seems to be a more difficult fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: Add test to verify data-loss bug was fixed inside the producer [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3498830886
CI check failed due to a failing test. Rebased with latest trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3413773317
I think it would be good the merge this PR. The unit test is for sure the first line of defense, but I believe the integration test would still provide value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3141537006
@mjsax -- I created a new PR to add the new unit tests as suggested.
Here is the new PR - https://github.com/apache/kafka/pull/20285
More details about the tests are described in the new PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3349787290
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3368546108
@junrao @shashankhs11 -- let's use https://github.com/apache/kafka/pull/20285 to fix the producer bug? Can you help reviewing Jun?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on code in PR #20254:
URL: https://github.com/apache/kafka/pull/20254#discussion_r2442692708
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.Sender;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.DEFAULT_TIMEOUT;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.apache.kafka.test.TestUtils.consumerConfig;
+import static org.apache.kafka.test.TestUtils.producerConfig;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Intended to reproduce KAFKA-19479
+ */
+@Timeout(600)
+@Tag("integration")
+public class AtLeastOnceDeliveryMessageLossIntegrationTest {
+ private static final Logger log = LoggerFactory.getLogger(
+ AtLeastOnceDeliveryMessageLossIntegrationTest.class);
+
+ private static final int NUM_BROKERS = 1;
+ private static final int LARGE_RECORD_COUNT = 50000;
+ private static final int SMALL_RECORD_COUNT = 40000;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @BeforeAll
+ public static void startCluster() throws IOException {
+ CLUSTER.start();
+ }
+
+ @AfterAll
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ private String applicationId;
+ private String inputTopic;
+ private String outputTopic;
+ private Properties streamsConfiguration;
+ private KafkaStreams kafkaStreams;
+
+ @BeforeEach
+ public void setUp(final TestInfo testInfo) throws Exception {
+ final String testId = safeUniqueTestName(testInfo);
+ applicationId = "app-" + testId;
+ inputTopic = "input-" + testId;
+ outputTopic = "output-" + testId;
+
+ cleanStateBeforeTest(CLUSTER, inputTopic, outputTopic);
+ CLUSTER.createTopics(inputTopic, outputTopic);
+
+ setupStreamsConfiguration();
+ }
+
+ @AfterEach
+ public void cleanUp() throws Exception {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30)); // need to stop due to infinite retries
+ }
+ if (streamsConfiguration != null) {
+ purgeLocalStreamsState(streamsConfiguration);
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceOutputRecordsWhenProducerFailsWithMessageTooLarge() throws Exception {
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(Sender.class)) {
+ produceInputData(LARGE_RECORD_COUNT);
+
+ kafkaStreams = createStreamsApplication();
+ startApplicationAndWaitUntilRunning(Collections.singletonList(kafkaStreams), Duration.ofMillis(DEFAULT_TIMEOUT));
+
+ waitForProcessingAndCommit();
+
+ assertTrue(appender.getMessages().stream()
+ .anyMatch(msg -> msg.contains("MESSAGE_TOO_LARGE") && msg.contains("splitting and retrying")),
+ "Should log MESSAGE_TOO_LARGE and splitting retry messages");
+
+ final int outputRecordCount = verifyOutputRecords(LARGE_RECORD_COUNT); // should produce records
+ final boolean offsetsCommitted = verifyConsumerOffsetsCommitted(LARGE_RECORD_COUNT); // should commit offset
+
+ assertEquals(LARGE_RECORD_COUNT, outputRecordCount, "Output topic should have " + LARGE_RECORD_COUNT + " records");
+ assertTrue(offsetsCommitted, "Consumer offsets should be committed");
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceMessagesNormallyForSmallerRecordCount() throws Exception {
Review Comment:
I am wondering if we need the 2nd test
Each test takes about 30-31s to run
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.Sender;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.DEFAULT_TIMEOUT;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.apache.kafka.test.TestUtils.consumerConfig;
+import static org.apache.kafka.test.TestUtils.producerConfig;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Intended to reproduce KAFKA-19479
+ */
+@Timeout(600)
+@Tag("integration")
+public class AtLeastOnceDeliveryMessageLossIntegrationTest {
+ private static final Logger log = LoggerFactory.getLogger(
+ AtLeastOnceDeliveryMessageLossIntegrationTest.class);
+
+ private static final int NUM_BROKERS = 1;
+ private static final int LARGE_RECORD_COUNT = 50000;
+ private static final int SMALL_RECORD_COUNT = 40000;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @BeforeAll
+ public static void startCluster() throws IOException {
+ CLUSTER.start();
+ }
+
+ @AfterAll
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ private String applicationId;
+ private String inputTopic;
+ private String outputTopic;
+ private Properties streamsConfiguration;
+ private KafkaStreams kafkaStreams;
+
+ @BeforeEach
+ public void setUp(final TestInfo testInfo) throws Exception {
+ final String testId = safeUniqueTestName(testInfo);
+ applicationId = "app-" + testId;
+ inputTopic = "input-" + testId;
+ outputTopic = "output-" + testId;
+
+ cleanStateBeforeTest(CLUSTER, inputTopic, outputTopic);
+ CLUSTER.createTopics(inputTopic, outputTopic);
+
+ setupStreamsConfiguration();
+ }
+
+ @AfterEach
+ public void cleanUp() throws Exception {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30)); // need to stop due to infinite retries
+ }
+ if (streamsConfiguration != null) {
+ purgeLocalStreamsState(streamsConfiguration);
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceOutputRecordsWhenProducerFailsWithMessageTooLarge() throws Exception {
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(Sender.class)) {
+ produceInputData(LARGE_RECORD_COUNT);
+
+ kafkaStreams = createStreamsApplication();
+ startApplicationAndWaitUntilRunning(Collections.singletonList(kafkaStreams), Duration.ofMillis(DEFAULT_TIMEOUT));
+
+ waitForProcessingAndCommit();
+
+ assertTrue(appender.getMessages().stream()
+ .anyMatch(msg -> msg.contains("MESSAGE_TOO_LARGE") && msg.contains("splitting and retrying")),
+ "Should log MESSAGE_TOO_LARGE and splitting retry messages");
+
+ final int outputRecordCount = verifyOutputRecords(LARGE_RECORD_COUNT); // should produce records
+ final boolean offsetsCommitted = verifyConsumerOffsetsCommitted(LARGE_RECORD_COUNT); // should commit offset
+
+ assertEquals(LARGE_RECORD_COUNT, outputRecordCount, "Output topic should have " + LARGE_RECORD_COUNT + " records");
+ assertTrue(offsetsCommitted, "Consumer offsets should be committed");
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceMessagesNormallyForSmallerRecordCount() throws Exception {
Review Comment:
I am wondering if we need the 2nd test?
Each test takes about 30-31s to run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3344291319
> Did you actually observe that flush() returns early?
@junrao Yes (as described above: https://github.com/apache/kafka/pull/20254#issuecomment-3134139853), I did set a breakpoint somewhere in the producer to see what it's doing, and while it was retrying the send internally it did unblock `flush()` before the send was done, and `producer.flush()` call did return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3368526760
Thanks @junrao!
> Does KStreams depend on Producer.flush() failing? It seems that KStreams needs to check the future returned in the send() call to know if the record is sent successfully.
We do not depend on `flush` to fail, but to block. We do check the send `Callback` (not the `Future`, but it's effectively the same thing, right?) after `flush` returned, but if `flush` does not block, the callbacks are not executed yet, and thus our check if any callback failed does not detect the error.
> Do you have a test that can reproduce this?
This PR add a test which reproduces the issue.
> This is indeed a bug in the producer. ProducerBatch has a produceFuture of type ProduceRequestResult and a list of Thunk, one for each produced record. Each Thunk has a future of type FutureRecordMetadata. When we split a ProducerBatch, we have the logic to chain Thunk.future to the newly split batches. This makes sure that we don't unblock the producer.send() prematurely. But the problem is that flush() doesn't wait on Thunk.future. Instead, it waits on ProducerBatch.produceFuture. After splitting, we immediately call ProducerBatch.produceFuture.done(). This will unblock the flush() call prematurely since the splitted batches haven't been completed.
Thanks for confirming. That matches my understanding.
Thanks for proposing a fix. I don't know the producer code well enough to propose one :)
> As for why the splitting loops forever, this is probably because of a recently fixed bug in https://github.com/apache/kafka/pull/20358/files.
Interesting. If we make `flush()` block as it should, and not let is exit early, we should hit `max.block.ms` inside `flush()` and it would throw a `TimeoutException` back into KS code? KS should handle this (or at least just crash). We can look into this as a follow up how a `TimeoutException` is handled for this case, and if KS does the right thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on code in PR #20254:
URL: https://github.com/apache/kafka/pull/20254#discussion_r2449290943
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.Sender;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.DEFAULT_TIMEOUT;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.apache.kafka.test.TestUtils.consumerConfig;
+import static org.apache.kafka.test.TestUtils.producerConfig;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Intended to reproduce KAFKA-19479
+ */
+@Timeout(600)
+@Tag("integration")
+public class AtLeastOnceDeliveryMessageLossIntegrationTest {
+ private static final Logger log = LoggerFactory.getLogger(
+ AtLeastOnceDeliveryMessageLossIntegrationTest.class);
+
+ private static final int NUM_BROKERS = 1;
+ private static final int LARGE_RECORD_COUNT = 50000;
+ private static final int SMALL_RECORD_COUNT = 40000;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @BeforeAll
+ public static void startCluster() throws IOException {
+ CLUSTER.start();
+ }
+
+ @AfterAll
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ private String applicationId;
+ private String inputTopic;
+ private String outputTopic;
+ private Properties streamsConfiguration;
+ private KafkaStreams kafkaStreams;
+
+ @BeforeEach
+ public void setUp(final TestInfo testInfo) throws Exception {
+ final String testId = safeUniqueTestName(testInfo);
+ applicationId = "app-" + testId;
+ inputTopic = "input-" + testId;
+ outputTopic = "output-" + testId;
+
+ cleanStateBeforeTest(CLUSTER, inputTopic, outputTopic);
+ CLUSTER.createTopics(inputTopic, outputTopic);
+
+ setupStreamsConfiguration();
+ }
+
+ @AfterEach
+ public void cleanUp() throws Exception {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30)); // need to stop due to infinite retries
+ }
+ if (streamsConfiguration != null) {
+ purgeLocalStreamsState(streamsConfiguration);
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceOutputRecordsWhenProducerFailsWithMessageTooLarge() throws Exception {
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(Sender.class)) {
+ produceInputData(LARGE_RECORD_COUNT);
+
+ kafkaStreams = createStreamsApplication();
+ startApplicationAndWaitUntilRunning(Collections.singletonList(kafkaStreams), Duration.ofMillis(DEFAULT_TIMEOUT));
+
+ waitForProcessingAndCommit();
+
+ assertTrue(appender.getMessages().stream()
+ .anyMatch(msg -> msg.contains("MESSAGE_TOO_LARGE") && msg.contains("splitting and retrying")),
+ "Should log MESSAGE_TOO_LARGE and splitting retry messages");
+
+ final int outputRecordCount = verifyOutputRecords(LARGE_RECORD_COUNT); // should produce records
+ final boolean offsetsCommitted = verifyConsumerOffsetsCommitted(LARGE_RECORD_COUNT); // should commit offset
+
+ assertEquals(LARGE_RECORD_COUNT, outputRecordCount, "Output topic should have " + LARGE_RECORD_COUNT + " records");
+ assertTrue(offsetsCommitted, "Consumer offsets should be committed");
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceMessagesNormallyForSmallerRecordCount() throws Exception {
Review Comment:
done, removed in bec0341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on code in PR #20254:
URL: https://github.com/apache/kafka/pull/20254#discussion_r2442692708
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.Sender;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.DEFAULT_TIMEOUT;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.apache.kafka.test.TestUtils.consumerConfig;
+import static org.apache.kafka.test.TestUtils.producerConfig;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Intended to reproduce KAFKA-19479
+ */
+@Timeout(600)
+@Tag("integration")
+public class AtLeastOnceDeliveryMessageLossIntegrationTest {
+ private static final Logger log = LoggerFactory.getLogger(
+ AtLeastOnceDeliveryMessageLossIntegrationTest.class);
+
+ private static final int NUM_BROKERS = 1;
+ private static final int LARGE_RECORD_COUNT = 50000;
+ private static final int SMALL_RECORD_COUNT = 40000;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @BeforeAll
+ public static void startCluster() throws IOException {
+ CLUSTER.start();
+ }
+
+ @AfterAll
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ private String applicationId;
+ private String inputTopic;
+ private String outputTopic;
+ private Properties streamsConfiguration;
+ private KafkaStreams kafkaStreams;
+
+ @BeforeEach
+ public void setUp(final TestInfo testInfo) throws Exception {
+ final String testId = safeUniqueTestName(testInfo);
+ applicationId = "app-" + testId;
+ inputTopic = "input-" + testId;
+ outputTopic = "output-" + testId;
+
+ cleanStateBeforeTest(CLUSTER, inputTopic, outputTopic);
+ CLUSTER.createTopics(inputTopic, outputTopic);
+
+ setupStreamsConfiguration();
+ }
+
+ @AfterEach
+ public void cleanUp() throws Exception {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30)); // need to stop due to infinite retries
+ }
+ if (streamsConfiguration != null) {
+ purgeLocalStreamsState(streamsConfiguration);
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceOutputRecordsWhenProducerFailsWithMessageTooLarge() throws Exception {
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(Sender.class)) {
+ produceInputData(LARGE_RECORD_COUNT);
+
+ kafkaStreams = createStreamsApplication();
+ startApplicationAndWaitUntilRunning(Collections.singletonList(kafkaStreams), Duration.ofMillis(DEFAULT_TIMEOUT));
+
+ waitForProcessingAndCommit();
+
+ assertTrue(appender.getMessages().stream()
+ .anyMatch(msg -> msg.contains("MESSAGE_TOO_LARGE") && msg.contains("splitting and retrying")),
+ "Should log MESSAGE_TOO_LARGE and splitting retry messages");
+
+ final int outputRecordCount = verifyOutputRecords(LARGE_RECORD_COUNT); // should produce records
+ final boolean offsetsCommitted = verifyConsumerOffsetsCommitted(LARGE_RECORD_COUNT); // should commit offset
+
+ assertEquals(LARGE_RECORD_COUNT, outputRecordCount, "Output topic should have " + LARGE_RECORD_COUNT + " records");
+ assertTrue(offsetsCommitted, "Consumer offsets should be committed");
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceMessagesNormallyForSmallerRecordCount() throws Exception {
Review Comment:
Would this test still be required?
Each test takes about 30-31s to run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3349188285
@mjsax
> Expected behavior would be, that producer.flush() blocks, and eventual fails with a TimeoutException hitting delivery.timeout.ms.
If a record fails to be delivered after delivery.timeout.ms, Sender calls failBatch(), which eventually calls ProduceRequestResult.done(). This unblocks Producer.flush(), but it won't fail with a TimeoutException since it doesn't check the error in ProduceRequestResult. Does KStreams depend on Producer.flush() failing? It seems that KStreams needs to check the future returned in the send() call to know if the record is sent successfully.
> Or is this by design, and if yes, why?), as there could also be the case of a single large message that cannot be split.
We have the following code in Sender. It only tries to split the batch if it has more than 1 record. So, the splitting shouldn't occur forever. It does have the issue that when a split batch is created, it gets a new createMs, which inadvently extends the timeout.
```
private void completeBatch(ProducerBatch batch, ProduceResponse.PartitionResponse response, long correlationId,
long now, Map<TopicPartition, Metadata.LeaderIdAndEpoch> partitionsWithUpdatedLeaderInfo) {
Errors error = response.error;
if (error == Errors.MESSAGE_TOO_LARGE && batch.recordCount > 1 && !batch.isDone() &&
(batch.magic() >= RecordBatch.MAGIC_VALUE_V2 || batch.isCompressed())) {
...
this.accumulator.splitAndReenqueue(batch);
```
> I did set a breakpoint somewhere in the producer to see what it's doing, and while it was retrying the send internally it did unblock flush() before the send was done, and producer.flush() call did return.
Do you have a test that can reproduce this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3363498212
@mjsax : Took a closer look. This is indeed a bug in the producer. ProducerBatch has a produceFuture of type ProduceRequestResult and a list of Thunk, one for each produced record. Each Thunk has a future of type FutureRecordMetadata. When we split a ProducerBatch, we have the logic to chain Thunk.future to the newly split batches. This makes sure that we don't unblock the producer.send() prematurely. But the problem is that flush() doesn't wait on Thunk.future. Instead, it waits on ProducerBatch.produceFuture. After splitting, we immediately call ProducerBatch.produceFuture.done(). This will unblock the flush() call prematurely since the splitted batches haven't been completed.
As for the fix, one way is to change the logic in RecordAccumulator.awaitFlushCompletion(). Instead of doing the following,
```
for (ProduceRequestResult result : this.incomplete.requestResults())
result.await();
```
we collect all thunks in each incomplete batch and wait on each thunk's FutureRecordMetadata. This way, the chaining logic for FutureRecordMetadata will kick in.
As for why the splitting loops forever, this is probably because of a recently fixed bug in https://github.com/apache/kafka/pull/20358/files.
@shashankhs11 : Would you be willing to submit a patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3373191390
> We do check the send Callback (not the Future, but it's effectively the same thing, right?) after flush returned, but if flush does not block, the callbacks are not executed yet, and thus our check if any callback failed does not detect the error.
Yes, waiting for the callback is the same thing. I guess you could design the callback to be more defensive by making sure that the callback is indeed called. In this bug, flush() does return before the callback is called.
> If we make flush() block as it should, and not let is exit early, we should hit max.block.ms inside flush() and it would throw a TimeoutException back into KS code?
flush() doesn't time out after max.block.ms. It just waits for all requests to be completed, which should happen after max delivery time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3136300976
Hey all, interesting. I don't have too much context but it does look fishy when we `tryAppendForSplit` and always allow a first record (no matter the size it seems), here https://github.com/apache/kafka/blob/4c1b79851f6858f42ca0eab33afa88af1c531c1f/clients/src/main/java/org/apache/kafka/common/record/MemoryRecordsBuilder.java#L856
Definitely needs more investigation, maybe start by adding a test to check the retry behaviour around `splitAndReenqueue` for a large batch with a single record? I cannot find any test for it (here it's all about batches that can indeed be split
https://github.com/apache/kafka/blob/4c1b79851f6858f42ca0eab33afa88af1c531c1f/clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java#L1016 )
Hope it helps, I will stay in the loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3261020620
@lianetm @junrao @mjsax tagging as a reminder for follow-up. Since, this is confirmed to be a bug (potentially critical) and I believe may require more discussions within the community experts.
To make it easier for others to reproduce and debug, one option could be that I can temporarily disable the failing test and we can merge this PR, so that community members can pick it up more easily.
Please let me know if I can contribute in any way to this issue. If there's any way I can help and contribute to this issue, I would really love to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: Add test to verify data-loss bug was fixed inside the producer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3499976282
Thanks for the test! Merged to `trunk`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3413236716
@mjsax @junrao
Bug was fixed in #20285. We should be good to close this PR as we decided to omit the integration test as per https://github.com/apache/kafka/pull/20285#discussion_r2434292455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #20254:
URL: https://github.com/apache/kafka/pull/20254#discussion_r2446756080
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/AtLeastOnceDeliveryMessageLossIntegrationTest.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.internals.Sender;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.DEFAULT_TIMEOUT;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.apache.kafka.test.TestUtils.consumerConfig;
+import static org.apache.kafka.test.TestUtils.producerConfig;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Intended to reproduce KAFKA-19479
+ */
+@Timeout(600)
+@Tag("integration")
+public class AtLeastOnceDeliveryMessageLossIntegrationTest {
+ private static final Logger log = LoggerFactory.getLogger(
+ AtLeastOnceDeliveryMessageLossIntegrationTest.class);
+
+ private static final int NUM_BROKERS = 1;
+ private static final int LARGE_RECORD_COUNT = 50000;
+ private static final int SMALL_RECORD_COUNT = 40000;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ @BeforeAll
+ public static void startCluster() throws IOException {
+ CLUSTER.start();
+ }
+
+ @AfterAll
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ private String applicationId;
+ private String inputTopic;
+ private String outputTopic;
+ private Properties streamsConfiguration;
+ private KafkaStreams kafkaStreams;
+
+ @BeforeEach
+ public void setUp(final TestInfo testInfo) throws Exception {
+ final String testId = safeUniqueTestName(testInfo);
+ applicationId = "app-" + testId;
+ inputTopic = "input-" + testId;
+ outputTopic = "output-" + testId;
+
+ cleanStateBeforeTest(CLUSTER, inputTopic, outputTopic);
+ CLUSTER.createTopics(inputTopic, outputTopic);
+
+ setupStreamsConfiguration();
+ }
+
+ @AfterEach
+ public void cleanUp() throws Exception {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30)); // need to stop due to infinite retries
+ }
+ if (streamsConfiguration != null) {
+ purgeLocalStreamsState(streamsConfiguration);
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceOutputRecordsWhenProducerFailsWithMessageTooLarge() throws Exception {
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(Sender.class)) {
+ produceInputData(LARGE_RECORD_COUNT);
+
+ kafkaStreams = createStreamsApplication();
+ startApplicationAndWaitUntilRunning(Collections.singletonList(kafkaStreams), Duration.ofMillis(DEFAULT_TIMEOUT));
+
+ waitForProcessingAndCommit();
+
+ assertTrue(appender.getMessages().stream()
+ .anyMatch(msg -> msg.contains("MESSAGE_TOO_LARGE") && msg.contains("splitting and retrying")),
+ "Should log MESSAGE_TOO_LARGE and splitting retry messages");
+
+ final int outputRecordCount = verifyOutputRecords(LARGE_RECORD_COUNT); // should produce records
+ final boolean offsetsCommitted = verifyConsumerOffsetsCommitted(LARGE_RECORD_COUNT); // should commit offset
+
+ assertEquals(LARGE_RECORD_COUNT, outputRecordCount, "Output topic should have " + LARGE_RECORD_COUNT + " records");
+ assertTrue(offsetsCommitted, "Consumer offsets should be committed");
+ }
+ }
+
+ @Test
+ public void shouldCommitOffsetsAndProduceMessagesNormallyForSmallerRecordCount() throws Exception {
Review Comment:
I agree. Having the negative test should be sufficient, as it's the interesting case which did not work. The positive case is implicitly covered by many other integration tests, so it seems unncessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3379422509
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3417752039
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3419083184
@mjsax I updated the test. Requesting to take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3134139853
Thanks for the PR. -- I did check it out and run it locally, and believe it's actually a bug inside `KafkaProducer`.
Before Kafka Streams commits offsets, it does call `producer.flush()` but this call does not block as it should until all requests are sent, but it returns early. -- Expected behavior would be, that `producer.flush()` blocks, and eventual fails with a `TimeoutException` hitting `delivery.timeout.ms`.
The problem inside the producer seems to be, that it initializes a flush by bumping `RecordAccumulator.flushesInProgress` -- when the "message too large" error return, it tries to split the batch, and does make the original (too large) batch as "done" which decrements the `flushesInProgress` counter to zero (leading to the early return of `flush()` which believes the flush is completed), what should not happen, as we are still in the middle of the flush, as new (smaller) batches get enqueued.
There seems to be some other issue in splitting batches though, as there is repeated logs:
```
WARN [Producer clientId=app-shouldNotCommitOffsetsAndNotProduceOutputRecordsWhenProducerFailsWithMessageTooLargewWTlaRhOS_68DNGthXrNsQ-d363a0a7-08d4-4f0d-b5f1-3347745661be-StreamThread-1-producer] Got error produce response in correlation id 47 on topic-partition output-shouldNotCommitOffsetsAndNotProduceOutputRecordsWhenProducerFailsWithMessageTooLargewWTlaRhOS_68DNGthXrNsQ-0, splitting and retrying (2147483647 attempts left). Error: MESSAGE_TOO_LARGE (org.apache.kafka.clients.producer.internals.Sender:667)
```
It seems, the producer is actually not able to split the batch, what I don't understand. Record size is small in the test setup, and batch size is set to 32MB... Also the counter `splitting and retrying (2147483647 attempts left)` does not decrease (not sure if this is right or wrong -- it seems, when we split a batch, and retry, the new batch(es) get a fresh retry count, what might be correct, but also somehow seems off?) -- This not being able to split a batch, is just a side issue, as there could also be the case of a single large message that cannot be split.
I am not a producer expert, and thus, I am not 100% sure right now what the right fix should be -- it seems to be a more difficult fix. \cc @lianetm @kirktrue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "shashankhs11 (via GitHub)" <gi...@apache.org>.
shashankhs11 commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3369655086
> let's use #20285 to fix the producer bug?
Done! I cherry-picked the commit to #20285 along with the integration test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19479: at_least_once mode in Kafka Streams silently drops messages when the producer fails with MESSAGE_TOO_LARGE, violating delivery guarantees [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20254:
URL: https://github.com/apache/kafka/pull/20254#issuecomment-3367813966
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org