You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mjsax (via GitHub)" <gi...@apache.org> on 2024/11/07 02:12:37 UTC

[PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

mjsax opened a new pull request, #17710:
URL: https://github.com/apache/kafka/pull/17710

   TimestampExtractor allows to drop records by returning a timestamp of -1. For this case, we still need to update consumed offsets to allows us to commit progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #17710:
URL: https://github.com/apache/kafka/pull/17710


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831959170


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -2156,7 +2194,6 @@ public void shouldCheckpointOffsetsOnPostCommit() {
         when(stateManager.taskId()).thenReturn(taskId);
         when(stateManager.taskType()).thenReturn(TaskType.ACTIVE);
         final long offset = 543L;
-        final long consumedOffset = 345L;

Review Comment:
   unused -- removing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831960504


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -248,18 +251,54 @@ private static StreamsConfig createConfig(final String enforcedProcessingValue)
     }
 
     private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue) {
-        return createConfig(eosConfig, enforcedProcessingValue, LogAndFailExceptionHandler.class.getName(), LogAndFailProcessingExceptionHandler.class.getName());
+        return createConfig(
+            eosConfig,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfig(final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",
+            deserializationExceptionHandler,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfigWithTsExtractor(final Class<? extends TimestampExtractor> timestampExtractor) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            timestampExtractor
+        );
     }
 
-    private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue, final String deserializationExceptionHandler) {
-        return createConfig(eosConfig, enforcedProcessingValue, deserializationExceptionHandler, LogAndFailProcessingExceptionHandler.class.getName());
+    private static StreamsConfig createConfig(
+        final String enforcedProcessingValue,
+        final Class<? extends ProcessingExceptionHandler> processingExceptionHandler
+    ) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            processingExceptionHandler,
+            FailOnInvalidTimestamp.class
+        );
     }
 
     private static StreamsConfig createConfig(
         final String eosConfig,
         final String enforcedProcessingValue,
-        final String deserializationExceptionHandler,
-        final String processingExceptionHandler) {
+        final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler,
+        final Class<? extends ProcessingExceptionHandler> processingExceptionHandler,
+        final Class<? extends TimestampExtractor> timestampExtractor) {

Review Comment:
   Adding new `TimestampExtractor` parameter here, and cleanup the overload of  `createConfig` a little bit, to avoid passing in unnecessary parameters (create a little noise on the PR below, but code is much cleaner now)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #17710:
URL: https://github.com/apache/kafka/pull/17710#issuecomment-2466555064

   Merged to `trunk` and cherry-picked to `3.9`,  `3.8`, and `3.7` branches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831959021


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -1451,18 +1492,15 @@ public void shouldBeProcessableIfAllPartitionsBuffered() {
 
         assertFalse(task.process(0L));
 
-        final byte[] bytes = ByteBuffer.allocate(4).putInt(1).array();
-
-        task.addRecords(partition1, singleton(new ConsumerRecord<>(topic1, 1, 0, bytes, bytes)));
+        task.addRecords(partition1, singleton(getConsumerRecordWithOffsetAsTimestamp(partition1, 0)));

Review Comment:
   need to change this to avoid a default ts which is negative and would break this test (same below)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1833344761


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -2773,10 +2894,8 @@ public void punctuateShouldThrowStreamsExceptionWhenProcessingExceptionHandlerRe
         when(stateManager.taskId()).thenReturn(taskId);
         when(stateManager.taskType()).thenReturn(TaskType.ACTIVE);
         task = createStatelessTask(createConfig(
-            AT_LEAST_ONCE,
             "100",
-            LogAndFailExceptionHandler.class.getName(),

Review Comment:
   Why remove here?



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -248,18 +251,54 @@ private static StreamsConfig createConfig(final String enforcedProcessingValue)
     }
 
     private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue) {
-        return createConfig(eosConfig, enforcedProcessingValue, LogAndFailExceptionHandler.class.getName(), LogAndFailProcessingExceptionHandler.class.getName());
+        return createConfig(
+            eosConfig,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfig(final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",
+            deserializationExceptionHandler,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfigWithTsExtractor(final Class<? extends TimestampExtractor> timestampExtractor) {
+        return createConfig(
+            AT_LEAST_ONCE,

Review Comment:
   Also why creating configs with `AT_LEAST_ONCE` only?



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -248,18 +251,54 @@ private static StreamsConfig createConfig(final String enforcedProcessingValue)
     }
 
     private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue) {
-        return createConfig(eosConfig, enforcedProcessingValue, LogAndFailExceptionHandler.class.getName(), LogAndFailProcessingExceptionHandler.class.getName());
+        return createConfig(
+            eosConfig,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfig(final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",
+            deserializationExceptionHandler,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfigWithTsExtractor(final Class<? extends TimestampExtractor> timestampExtractor) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",

Review Comment:
   same as above



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -2795,10 +2914,8 @@ public void punctuateShouldThrowStreamsExceptionWhenProcessingExceptionHandlerRe
         when(stateManager.taskId()).thenReturn(taskId);
         when(stateManager.taskType()).thenReturn(TaskType.ACTIVE);
         task = createStatelessTask(createConfig(
-            AT_LEAST_ONCE,
             "100",
-            LogAndFailExceptionHandler.class.getName(),

Review Comment:
   same here and below



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -248,18 +251,54 @@ private static StreamsConfig createConfig(final String enforcedProcessingValue)
     }
 
     private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue) {
-        return createConfig(eosConfig, enforcedProcessingValue, LogAndFailExceptionHandler.class.getName(), LogAndFailProcessingExceptionHandler.class.getName());
+        return createConfig(
+            eosConfig,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfig(final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",

Review Comment:
   What is represented by the `"0"`? Maybe add a named variable to help grokking what's going on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831958025


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java:
##########
@@ -340,7 +348,7 @@ public void shouldSetTimestampAndRespectMaxTimestampPolicy() {
 
     @Test
     public void shouldThrowStreamsExceptionWhenKeyDeserializationFails() {
-        final byte[] key = Serdes.Long().serializer().serialize("foo", 1L);
+        final byte[] key = new LongSerializer().serialize("foo", 1L);

Review Comment:
   Side cleanup



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1833595738


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -248,18 +251,54 @@ private static StreamsConfig createConfig(final String enforcedProcessingValue)
     }
 
     private static StreamsConfig createConfig(final String eosConfig, final String enforcedProcessingValue) {
-        return createConfig(eosConfig, enforcedProcessingValue, LogAndFailExceptionHandler.class.getName(), LogAndFailProcessingExceptionHandler.class.getName());
+        return createConfig(
+            eosConfig,
+            enforcedProcessingValue,
+            LogAndFailExceptionHandler.class,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfig(final Class<? extends DeserializationExceptionHandler> deserializationExceptionHandler) {
+        return createConfig(
+            AT_LEAST_ONCE,
+            "0",
+            deserializationExceptionHandler,
+            LogAndFailProcessingExceptionHandler.class,
+            FailOnInvalidTimestamp.class
+        );
+    }
+
+    private static StreamsConfig createConfigWithTsExtractor(final Class<? extends TimestampExtractor> timestampExtractor) {
+        return createConfig(
+            AT_LEAST_ONCE,

Review Comment:
   Why would we want to run this test with EOS? This helper is only used for the new test of this PR. I don't see any reason why we should run the test with EOS? (In the end, the alos/eos config does not really matter, but given the helper overloads we have, we need to pass in something if we don't want to get too many overloads.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831957679


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java:
##########
@@ -240,6 +240,7 @@ private void updateHead() {
                         deserialized.topic(), deserialized.partition(), deserialized.offset(), timestamp, timestampExtractor.getClass().getCanonicalName()
                 );
                 droppedRecordsSensor.record();
+                lastCorruptedRecord = raw;

Review Comment:
   This is the actual fix -- we track the record with invalid ts as "corrupted record" instead of dropping it completely. The `CorrecuptedRecrods` allows us to update the offset later, but would be dropped by itself and not be processed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1833597228


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -2795,10 +2914,8 @@ public void punctuateShouldThrowStreamsExceptionWhenProcessingExceptionHandlerRe
         when(stateManager.taskId()).thenReturn(taskId);
         when(stateManager.taskType()).thenReturn(TaskType.ACTIVE);
         task = createStatelessTask(createConfig(
-            AT_LEAST_ONCE,
             "100",
-            LogAndFailExceptionHandler.class.getName(),

Review Comment:
   Same answer as above,
   
   If helpful, happy to split out the refactoring into it's own PR (or maybe even multiple to make it simpler to follow -- it's a little messy...)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1833596691


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -2773,10 +2894,8 @@ public void punctuateShouldThrowStreamsExceptionWhenProcessingExceptionHandlerRe
         when(stateManager.taskId()).thenReturn(taskId);
         when(stateManager.taskType()).thenReturn(TaskType.ACTIVE);
         task = createStatelessTask(createConfig(
-            AT_LEAST_ONCE,
             "100",
-            LogAndFailExceptionHandler.class.getName(),

Review Comment:
   Just simplifying the setup: added a new `createConfig` overload which does set `LogAndFailExceptionHandler` by default (instead of using the old one which always get `LogAndFailExceptionHandler` passed when used)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831957849


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordQueueTest.java:
##########
@@ -74,8 +74,7 @@ public class RecordQueueTest {
     private final StreamsMetricsImpl streamsMetrics =
         new StreamsMetricsImpl(metrics, "mock", new MockTime());
 
-    @SuppressWarnings("rawtypes")
-    final InternalMockProcessorContext context = new InternalMockProcessorContext<>(
+    final InternalMockProcessorContext<Integer, Integer> context = new InternalMockProcessorContext<>(

Review Comment:
   Side cleanup to fix warnings



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17872: Update consumed offsets on records with invalid timestamp [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #17710:
URL: https://github.com/apache/kafka/pull/17710#discussion_r1831958584


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -400,36 +440,38 @@ public void shouldAutoOffsetResetIfNoCommittedOffsetFound() {
         task.addPartitionsForOffsetReset(Collections.singleton(partition1));
 
         final AtomicReference<AssertionError> shouldNotSeek = new AtomicReference<>();
-        final MockConsumer<byte[], byte[]> consumer = new MockConsumer<byte[], byte[]>(OffsetResetStrategy.EARLIEST) {
-            @Override
-            public void seek(final TopicPartition partition, final long offset) {
-                final AssertionError error = shouldNotSeek.get();
-                if (error != null) {
-                    throw error;
+        try (final MockConsumer<byte[], byte[]> consumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST) {

Review Comment:
   side cleanup



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org