[jira] [Commented] (KAFKA-20876) Windowed restore optimisation gives up when `endOffset-1` is a transaction control record

["Yunseop Eom (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101109#comment-18101109 ] 

Yunseop Eom commented on KAFKA-20876:
-------------------------------------

Opened PR #23029: https://github.com/apache/kafka/pull/23029

The windowed changelog restore optimization previously probed only endOffset - 1. When that offset is a transaction control record, the consumer filters it from poll results and the optimization incorrectly falls back to restoring from the beginning.

The PR retries from the preceding offset before falling back. TDD verification reproduced the failure on the baseline branch and passed after the implementation change. The affected StoreChangelogReaderTest, streams checkstyle, SpotBugs, and git diff --check pass locally. The PR is open and awaiting review.

> Windowed restore optimisation gives up when `endOffset-1` is a transaction control record
> -----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-20876
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20876
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.4.0
>            Reporter: Alan Lau
>            Assignee: Alan Lau
>            Priority: Blocker
>
> When restoring a windowed store with no checkpoint, {{StoreChangelogReader}} seeks to {{latestTimestamp - retentionPeriod}} instead of {{{}log-start{}}}, since data older than the retention is discarded on write. It learns {{latestTimestamp}} by seeking to {{endOffset - 1}} and polling once in {{{}seekNewPartitions{}}}; an empty poll falls back to {{{}seekToBeginning{}}}.
> On a 4.3 EOS-v2 soak, {{endOffset - 1}} returned no record on 195 of 195 probes. Dumping the segment confirms the last batch is a transaction control record (isControl: true, count: 1, size: 78), and the pattern recurs throughout — every transaction ends with a marker occupying an offset that is never delivered to a consumer. The poll therefore returns nothing and the optimisation is abandoned, leaving the restore at log-start with no margin against retention.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)