You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "shantanushukla (via GitHub)" <gi...@apache.org> on 2026/08/30 05:30:37 UTC

[PR] KAFKA-20913: Snapshot CompletedBatch appendTimestamp before buffer reuse [kafka]

shantanushukla opened a new pull request, #23311:
URL: https://github.com/apache/kafka/pull/23311

   JIRA: https://issues.apache.org/jira/browse/KAFKA-20913
   
   On the Raft leader, `KafkaRaftClient.appendBatch` writes the batch to the log, registers an async commit callback, then `release()`s the pooled buffer. `CompletedBatch.appendTimestamp()` was reading `MemoryRecords.firstBatch().maxTimestamp()` from a slice of that buffer. Production uses `BatchMemoryPool`, which requeues the buffer; a later batch overwrites the v2 header. When HWM later completes the callback, `handleCommit` can deliver the newer batch's append time.
   
   ### Fix
   Snapshot `maxTimestamp` when `CompletedBatch` is constructed and return that field from `appendTimestamp()`. The timestamp is fixed at build time (`BatchBuilder` writes `time.milliseconds()` into the header). Record payloads and `sizeInBytes()` are not affected.
   
   ### Why existing tests missed this
   - `RaftClientTestContext` defaults to `MemoryPool.NONE`, which allocates a new buffer every time and never reuses.
   - `BatchAccumulatorTest` called `appendTimestamp()` before `release()`, or verified `release()` without reading the timestamp afterward.
   - If the listener is still at the starting offset, commits come from log replay (the log copies records at append time), which hides the in-memory path.
   
   ### Tests
   These failed on trunk (first batch reported T2) and pass after the snapshot:
   
   - `BatchAccumulatorTest.testAppendTimestampUnchangedAfterBufferReleasedAndReused` — one-buffer `BatchMemoryPool`; drain/release at T1, append at T2; first batch must still report T1.
   - `KafkaRaftClientTest.testLeaderHandleCommitAppendTimestampStableWhenPoolReusesBuffer` — two voters so HWM lags; catch the listener up past the leader-change batch; append at T1 then T2 with a single reused buffer; after commit the listener's first data batch must still be T1.
   
   Related existing tests still pass: full `BatchAccumulatorTest`, `testListenerCommitCallbackAfterLeaderWrite`, `testAccumulatorClearedAfterBecomingFollower`.
   
   ```
   ./gradlew :raft:spotlessApply :raft:checkstyleMain :raft:checkstyleTest :raft:spotlessCheck
   ./gradlew :raft:test --tests org.apache.kafka.raft.internals.BatchAccumulatorTest --tests org.apache.kafka.raft.KafkaRaftClientTest.testLeaderHandleCommitAppendTimestampStableWhenPoolReusesBuffer --tests org.apache.kafka.raft.KafkaRaftClientTest.testListenerCommitCallbackAfterLeaderWrite --tests org.apache.kafka.raft.KafkaRaftClientTest.testAccumulatorClearedAfterBecomingFollower
   ```
   BUILD SUCCESSFUL.
   
   This work is original and is licensed to the Apache Kafka project under the Apache License, Version 2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20913: Snapshot CompletedBatch appendTimestamp before buffer reuse [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23311:
URL: https://github.com/apache/kafka/pull/23311#issuecomment-5564570005

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org