[jira] [Updated] (KAFKA-10249) In-memory stores are skipped when checkpointing but not skipped when reading the checkpoint

["John Roesler (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Roesler updated KAFKA-10249:
---------------------------------
    Fix Version/s: 2.6.0

> In-memory stores are skipped when checkpointing but not skipped when reading the checkpoint
> -------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-10249
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10249
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 2.6.0
>            Reporter: Sophie Blee-Goldman
>            Assignee: Sophie Blee-Goldman
>            Priority: Major
>             Fix For: 2.6.0
>
>
> As the title suggests, offsets for in-memory stores (including the suppression buffer) are not written to the checkpoint file. However, when reading from the checkpoint file during task initialization, we do not check StateStore#persistent. We attempt to look up the offsets for in-memory stores in the checkpoint file, and obviously do not find them.
> With eos we have to conclude that the existing state is dirty and thus throw a TaskCorruptedException. So pretty much any task with in-memory state will always hit this exception when reinitializing from the checkpoint, forcing it to clear the entire state directory and build up all of its state again from scratch (both persistent and in-memory).
> This is especially unfortunate for KIP-441, as we will hit this any time a task is moved from one thread to another.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)