You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Uladzislau Blok (Jira)" <ji...@apache.org> on 2025/10/09 09:11:00 UTC
[jira] [Commented] (KAFKA-19430) Don't fail on RecordCorruptedException
[ https://issues.apache.org/jira/browse/KAFKA-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028599#comment-18028599 ]
Uladzislau Blok commented on KAFKA-19430:
-----------------------------------------
Hello. I did more tests locally and it looks to me as this exception _could_ sometimes be retryable:
# There is corruption on the disk. In this case partition will fail with exception when trying to read log fail and return error response to consumer
# There is network issue when transfer message over internet. for more details you can check this https://issues.apache.org/jira/browse/KAFKA-19613
I have created KIP to expose TP and offset as part of this exception, but I still see few gaps. If we'll just skip corrupted part we can lose the messages between current offset and corrupted part, I believe this is not good
I can see new approach for to fix it (inspired by rust). How [~mjsax] said DeserializationExceptionHandler is the read-path handler and we can try to catch the exception when pooling the message and then propagate it to current handler.
Now:
{code:java}
ConsumerRecords<byte[], byte[]> records = ConsumerRecords.empty();
try {
records = mainConsumer.poll(pollTime);
} catch (final InvalidOffsetException e) {
resetOffsets(e.partitions(), e);
}
return records;
{code}
Proposal:
{code:java}
Result<ConsumerRecords<byte[], byte[]>, Exception> result = Result.of(ConsumerRecords.empty());
try {
result = result.of(mainConsumer.poll(pollTime));
} catch (final InvalidOffsetException e) {
resetOffsets(e.partitions(), e);
} catch (final Exception e) {
result = result.of(e);
}
return result; {code}
This way we can catch either records or exception and 'unpack' the result inside of exception handler.
[~mjsax] [~lianetm] Any thoughts?
> Don't fail on RecordCorruptedException
> --------------------------------------
>
> Key: KAFKA-19430
> URL: https://issues.apache.org/jira/browse/KAFKA-19430
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Uladzislau Blok
> Priority: Major
>
> From [https://github.com/confluentinc/kafka-streams-examples/issues/524]
> Currently, the existing `DeserializationExceptionHandler` is applied when de-serializing the record key/value byte[] inside Kafka Streams. This implies that a `RecordCorruptedException` is not handled.
> We should explore to not let Kafka Streams crash, but maybe retry this error automatically (as `RecordCorruptedException extends RetriableException`), and find a way to pump the error into the existing exception handler.
> If the error is transient, user can still use `REPLACE_THREAD` in the uncaught exception handler, but this is a rather heavy weight approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)