[PR] KAFKA-20876: Retry windowed restore after control record [kafka]

["lh0156 (via GitHub)" <gi...@apache.org>]

lh0156 opened a new pull request, #23029:
URL: https://github.com/apache/kafka/pull/23029

   ## Summary
   
   When a windowed changelog ends with a transaction control record, the restore consumer filters that record from poll results. Probing only `endOffset - 1` therefore returns no record and incorrectly abandons the retention-based restore optimization.
   
   This change retries the probe from the preceding offset before falling back to the beginning. The regression test covers both a visible latest record and the filtered control-record case.
   
   ## Testing
   
   - TDD RED: the new filtered-record case failed on the baseline branch because restoration fell back to the beginning.
   - TDD GREEN: `./gradlew :streams:test --tests org.apache.kafka.streams.processor.internals.StoreChangelogReaderTest.shouldSeekByTimestampForWindowedStoreWithoutCheckpoint --no-build-cache --console=plain`
   - The targeted Gradle run completed the affected streams checkstyle and SpotBugs tasks successfully.
   - `git diff --check`
   
   Jira: https://issues.apache.org/jira/browse/KAFKA-20876


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-20876: Retry windowed restore after control record [kafka]

["ChoMinGi (via GitHub)" <gi...@apache.org>]

ChoMinGi commented on PR #23029:
URL: https://github.com/apache/kafka/pull/23029#issuecomment-5169874191

   Thanks for working on this. I tried the probe sequence with `EmbeddedKafkaCluster` and a real `read_committed` consumer using `pollTime = 100 ms`. I saw two cases worth considering.
   
   First, the retry still returned empty:
   
   ```text
   Committed transaction: 5 records + COMMIT marker, LSO=6
   
   seek(5) -> 0 records,   2 ms
   seek(4) -> 0 records, 101 ms   <- retry
   ...
   record returned on poll #6, 508 ms in
   
   Control: fresh consumer, seek(4) -> 1 record in 2 ms
   ```
   
   Second, the `endOffset - 2` retry did not reach the latest visible record when the log ended with an aborted transaction:
   
   ```text
   Committed transaction followed by an aborted transaction, LSO=12
   
   seek(11) -> 0
   seek(10) -> 0   <- retry
   seek(9..5) -> 0
   seek(4)  -> 1
   ```
   
   Here, the latest visible record was eight offsets back because the aborted batch was filtered as a whole.
   
   Would it be better to start the probe from an earlier offset, clamped to `logStartOffset`, instead of retrying at `endOffset - 2`?
   
   I can share the reproduction test if needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-20876: Retry windowed restore after control record [kafka]

["lh0156 (via GitHub)" <gi...@apache.org>]

lh0156 commented on PR #23029:
URL: https://github.com/apache/kafka/pull/23029#issuecomment-5159440601

   Verification update: the full `:streams:test` suite passed locally in 6m 54s after the targeted RED/GREEN regression test and affected checkstyle/SpotBugs tasks also passed. The branch remains clean and is ready for maintainer review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-20876: Retry windowed restore after control record [kafka]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #23029:
URL: https://github.com/apache/kafka/pull/23029#issuecomment-5235602426

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org