You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/27 20:19:40 UTC

Re: [PR] [bugfix] KafkaPartitionLevelConsumer: stop re-seeking past read_committed-filtered batches [pinot]

xiangfu0 commented on PR #18337:
URL: https://github.com/apache/pinot/pull/18337#issuecomment-4330182267

   Addressed review comments in commit \`7a5d6f50be\`. Title and description updated to reflect this is a bug fix in \`KafkaPartitionLevelConsumer\`, not just a flake-fix.
   
   **Per @noob-se7en (top-level):** Title is now \"[bugfix] KafkaPartitionLevelConsumer: stop re-seeking past read_committed-filtered batches\". Description has a dedicated **Behaviour change** section calling out that \`offsetOfNextBatch\` returned by an empty fetch now reflects the consumer's actual position (was: always \`startOffset\`).
   
   **Per @xiangfu0 (line 133, the still-re-seeking bug):** You were correct — my previous fix set \`_lastFetchedOffset = currentPosition - 1\` after an empty poll, which made the seek-check \`_lastFetchedOffset != startOffset - 1\` fire on the next call (e.g. \`17762 != -1\`) because \`RealtimeSegmentDataManager._currentOffset\` doesn't advance on empty batches and the caller passes the same \`startOffset\` again. Replaced with a single \`_nextReadOffset\` field; the new seek-check is \`_nextReadOffset < 0 || startOffset > _nextReadOffset\`, which **does not** re-seek when \`startOffset <= _nextReadOffset\` (the empty-poll-while-caller-stalled case).
   
   **Per @noob-se7en (line 56, single-field suggestion):** Done — \`_lastFetchedOffset\` + \`_lastSeekedStartOffset\` collapsed into the single \`_nextReadOffset\` field per your suggestion. Cleaner, and removes the bug class above by construction (one field can't disagree with itself).
   
   **Per @xiangfu0 (line 114, setUp reorder masks the regression):** Reverted. The test now uses the inherited setUp ordering (\`addTableConfig\` → \`waitForAllRealtimePartitionsConsuming\` → \`pushAvroIntoKafka\` → \`waitForAllDocsLoaded\`), which is exactly the path that exposes the consumer's seek-during-push bug. With the consumer fix in this PR the test should converge; without it (regression) the test fails. So the test now actually covers the fix.
   
   CI is running.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org