You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "krishan1390 (via GitHub)" <gi...@apache.org> on 2026/08/25 09:16:52 UTC
[PR] Default Kafka consumer auto.offset.reset to earliest to avoid data loss on expired offsets [pinot]
krishan1390 opened a new pull request, #19359:
URL: https://github.com/apache/pinot/pull/19359
### Problem
The Kafka clients created for realtime ingestion never receive an `auto.offset.reset` value, so they fall back to Kafka's default of `latest`. Pinot seeks to an explicit start offset before every poll, so this policy only takes effect when that offset is no longer present in the log — which is exactly the case where it matters.
When retention deletes records a consuming segment has not read yet, the client silently repositions to the **end** of the log, dropping every record still retained between the expired offset and the log end. A table catching up on a backlog can lose an arbitrary amount of retained data this way, and the further behind it is, the more it loses.
Setting `stream.kafka.consumer.prop.auto.offset.reset` does not prevent this. Despite its name, that key is parsed into the table's initial-offset criteria (`OffsetCriteria`) and never reaches the Kafka client; its accepted values (`smallest` / `largest`) are not valid Kafka values either.
### Fix
Default the Kafka client property to `earliest`, so an expired start offset resumes from the oldest retained record and only the records the broker actually deleted are skipped. The resulting gap is still surfaced by the existing `StreamDataLoss` detection.
Setting the raw `auto.offset.reset` property in `streamConfigs` continues to take precedence, and the initial-offset criteria is unaffected — it still decides only where a partition with no prior offset starts consuming.
### Testing
`KafkaPartitionLevelConsumerTest#testOffsetsExpired` pinned `auto.offset.reset=earliest` directly in the stream config, which masked the bug. It now uses the configuration an operator would actually write: on master it returns 0 records where 500 are expected, and it passes with this change. It also asserts that the skipped records are reported as data loss.
Applied to both `pinot-kafka-3.0` and `pinot-kafka-4.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Default Kafka consumer auto.offset.reset to earliest to avoid data loss on expired offsets [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19359:
URL: https://github.com/apache/pinot/pull/19359#issuecomment-5436522452
Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/1012 (merged). It documents the Kafka client default of `auto.offset.reset=earliest` for expired stored offsets, the raw-property override, unavoidable data-loss reporting, and the distinction from Pinot initial-offset criteria.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Default Kafka consumer auto.offset.reset to earliest to avoid data loss on expired offsets [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps merged PR #19359:
URL: https://github.com/apache/pinot/pull/19359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Default Kafka consumer auto.offset.reset to earliest to avoid data loss on expired offsets [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19359:
URL: https://github.com/apache/pinot/pull/19359#issuecomment-5409247586
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19359?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 100.00%. Comparing base ([`4824a5f`](https://app.codecov.io/gh/apache/pinot/commit/4824a5fd1f793b55a02de69f00395497830cf7a7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`0f994e9`](https://app.codecov.io/gh/apache/pinot/commit/0f994e99559b5969458e04c9c311f2538c54b981?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19359 +/- ##
==============================================
+ Coverage 67.20% 100.00% +32.79%
+ Complexity 1418 6 -1412
==============================================
Files 3479 3 -3476
Lines 223075 6 -223069
Branches 35135 0 -35135
==============================================
- Hits 149920 6 -149914
+ Misses 61184 0 -61184
+ Partials 11971 0 -11971
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+100.00%)` | :arrow_up: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (?)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.79%)` | :arrow_up: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+100.00%)` | :arrow_up: |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.79%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19359/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19359?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org