[jira] [Updated] (KAFKA-20663) KIP-1035: stale persisted changelog offset causes OffsetOutOfRangeException/TaskCorruptedException on restart

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax updated KAFKA-20663:
------------------------------------
    Affects Version/s:     (was: 4.3.1)
                           (was: 4.4.0)

> KIP-1035: stale persisted changelog offset causes OffsetOutOfRangeException/TaskCorruptedException on restart
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-20663
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20663
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.3.0
>            Reporter: Bill Bejeck
>            Assignee: Bill Bejeck
>            Priority: Blocker
>             Fix For: 4.3.1
>
>
> In 4.3, KIP-1035 moved the changelog offset into RocksDB and removed the forced flush on commit, so the persisted offset is now only made durable by an organic memtable flush or a clean close. When that offset goes stale — after an unclean exit, or a clean shutdown followed by changelog truncation/compaction while the instance is down — and the changelog log-start offset has advanced past it, the restore consumer seeks out of range and throws OffsetOutOfRangeException, which Streams converts to a TaskCorruptedException (full local-state wipe and rebuild). This happens far more often than in 4.2 (where the forced flush kept the offset within roughly commit.interval.ms), affecting both at-least-once and exactly-once and hitting windowed/segmented stores hardest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)