[jira] [Commented] (KAFKA-21101) Drop the remaining records of a batch when encountering an invalid record

["Chia-Ping Tsai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115670#comment-18115670 ] 

Chia-Ping Tsai commented on KAFKA-21101:
----------------------------------------

Records exceeding max.decompressed.message.bytes are out of scope here. They are valid records, and KAFKA-21082 / KAFKA-21100 handle them without dropping anything. This issue only covers records the parser cannot read at all.

> Drop the remaining records of a batch when encountering an invalid record
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-21101
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21101
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Chia-Ping Tsai
>            Assignee: Chia-Ping Tsai
>            Priority: Major
>
> Currently, an invalid record marks the partition "uncleanable", which blocks the cleaner from compacting all other records in that partition. Since the record is already invalid regardless of the root cause, nobody can read it or anything after it in the same batch (consumers parse a batch sequentially and fail at the same point), so dropping them loses nothing observable. We should offer an option to drop the invalid record and the remaining records of its batch, and keep the partition cleanable. Records before the invalid one are parsed successfully and are retained as usual.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)