[jira] [Updated] (KAFKA-20876) Windowed restore optimisation gives up when `endOffset-1` is a transaction control record

["Ken Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ken Huang updated KAFKA-20876:
------------------------------
    Fix Version/s:     (was: 4.3.2)

> Windowed restore optimisation gives up when `endOffset-1` is a transaction control record
> -----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-20876
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20876
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.3.0, 4.4.0
>            Reporter: Alan Lau
>            Assignee: Alan Lau
>            Priority: Blocker
>             Fix For: 4.4.1
>
>
> When restoring a windowed store with no checkpoint, {{StoreChangelogReader}} seeks to {{latestTimestamp - retentionPeriod}} instead of {{{}log-start{}}}, since data older than the retention is discarded on write. It learns {{latestTimestamp}} by seeking to {{endOffset - 1}} and polling once in {{{}seekNewPartitions{}}}; an empty poll falls back to {{{}seekToBeginning{}}}.
> On a 4.3 EOS-v2 soak, {{endOffset - 1}} returned no record on 195 of 195 probes. Dumping the segment confirms the last batch is a transaction control record (isControl: true, count: 1, size: 78), and the pattern recurs throughout — every transaction ends with a marker occupying an offset that is never delivered to a consumer. The poll therefore returns nothing and the optimisation is abandoned, leaving the restore at log-start with no margin against retention.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)