[jira] [Created] (KAFKA-21367) Stale recovery checkpoint can overwrite truncation checkpoint and skip log recovery

["Xiaobing Fang (Jira)" <ji...@apache.org>]

Xiaobing Fang created KAFKA-21367:
-------------------------------------

             Summary: Stale recovery checkpoint can overwrite truncation checkpoint and skip log recovery
                 Key: KAFKA-21367
                 URL: https://issues.apache.org/jira/browse/KAFKA-21367
             Project: Kafka
          Issue Type: Bug
          Components: core
            Reporter: Xiaobing Fang


h3. Problem

{\{LogManager.checkpointRecoveryOffsetsInDir}} collects recovery points before calling \{{OffsetCheckpointFile.write}}. The lock in \{{CheckpointFile.write}} serializes file writes, but does not cover recovery point sampling.

A periodic checkpoint can therefore overwrite a checkpoint written after log truncation:

# A checkpoint task samples \{{recoveryPoint = 10}}, then pauses before writing.
# Another thread truncates the log to offset 0, lowering its recovery point to 0, and writes a checkpoint containing 0.
# The first task resumes and writes its previously sampled value of 10.

The persisted recovery point is now ahead of the actual flushed boundary. This affects checkpoints following both \{{truncateTo}} and \{{truncateFullyAndStartAt}}.

h3. Recovery impact

After truncation, new records can be appended and the log can roll before the asynchronous flush completes. If an unclean shutdown leaves a damaged batch in an earlier segment, the stale checkpoint can cause recovery to skip that segment.

For example, with segment base offsets \{0, 1} and a stale checkpoint of 10, \{{LogSegments.values(10, Long.MAX_VALUE)}} starts at \{{floorKey(10) = 1}}. Segment 0 is not recovered. With the correct checkpoint of 0, both segments are included.

A corrupt batch in the skipped segment can survive startup and fail subsequent record validation with \{{CorruptRecordException}}. The race does not itself corrupt record bytes; it can prevent recovery from detecting and truncating damaged records.

h3. Reproduction

The checkpoint overwrite was reproduced deterministically on trunk at \{{2bd1d20063ce7e0843bb1482f7252b67c2a40421}}. Tests for both truncation paths read back 10 from the checkpoint file instead of the expected 0.

The recovery consequence was separately reproduced using controlled CRC corruption, a segment roll, a pending asynchronous flush, and reopening the log without a clean-shutdown marker. With the stale checkpoint, the damaged earlier segment was skipped. With the correct checkpoint, recovery detected the corruption and truncated the log.

CRC corruption was injected to test the recovery behavior; this was not a real power-loss experiment.

h3. Proposed fix

Serialize recovery point sampling and checkpoint writing using the same per-directory checkpoint object. This prevents an older snapshot from being written after a newer truncation checkpoint, without changing the checkpoint format.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)