[jira] [Created] (KAFKA-10249) In-memory stores are skipped when checkpointing but not skipped when reading the checkpoint

["Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

Sophie Blee-Goldman created KAFKA-10249:
-------------------------------------------

             Summary: In-memory stores are skipped when checkpointing but not skipped when reading the checkpoint
                 Key: KAFKA-10249
                 URL: https://issues.apache.org/jira/browse/KAFKA-10249
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 2.6.0
            Reporter: Sophie Blee-Goldman
            Assignee: Sophie Blee-Goldman


As the title suggests, offsets for in-memory stores (including the suppression buffer) are not written to the checkpoint file. However, when reading from the checkpoint file during task initialization, we do not check StateStore#persistent. We attempt to look up the offsets for in-memory stores in the checkpoint file, and obviously do not find them.

With eos we have to conclude that the existing state is dirty and thus throw a TaskCorruptedException. So pretty much any task with in-memory state will always hit this exception when reinitializing from the checkpoint, forcing it to clear the entire state directory and build up all of its state again from scratch (both persistent and in-memory).

This is especially unfortunate for KIP-441, as we will hit this any time a task is moved from one thread to another.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)