You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "deepthi912 (via GitHub)" <gi...@apache.org> on 2025/11/05 00:58:36 UTC
[PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
deepthi912 opened a new pull request, #17144:
URL: https://github.com/apache/pinot/pull/17144
Continuation to : https://github.com/apache/pinot/pull/17139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on PR #17144:
URL: https://github.com/apache/pinot/pull/17144#issuecomment-3493161347
> mostly lgtm! can you add some tests @deepthi912 to ensure that this correctly rejects force commit for partial upserts but not for other table types?
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17144:
URL: https://github.com/apache/pinot/pull/17144#issuecomment-3488768711
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17144?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.05%. Comparing base ([`84505f8`](https://app.codecov.io/gh/apache/pinot/commit/84505f8070b57f9bae1799b7d73b04105b747e81?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`99c0fe5`](https://app.codecov.io/gh/apache/pinot/commit/99c0fe5613c7b43c363ac4faa3b0815c2b802962?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17144?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/17144?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2FBaseTableDataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17144?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (84505f8) and HEAD (99c0fe5). Click for more details.
>
> <details><summary>HEAD has 28 uploads less than BASE</summary>
>
>| Flag | BASE (84505f8) | HEAD (99c0fe5) |
>|------|------|------|
>|java-21|4|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|8|1|
>|java-11|4|0|
>|unittests2|2|0|
>|integration|4|0|
>|integration1|2|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17144 +/- ##
============================================
- Coverage 63.17% 56.05% -7.13%
+ Complexity 1428 701 -727
============================================
Files 3114 2429 -685
Lines 183908 135736 -48172
Branches 28167 21670 -6497
============================================
- Hits 116187 76085 -40102
+ Misses 58738 53257 -5481
+ Partials 8983 6394 -2589
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.05% <0.00%> (-7.09%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.05% <0.00%> (-7.13%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.05% <0.00%> (-7.12%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.05% <0.00%> (-0.06%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17144/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17144?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17144:
URL: https://github.com/apache/pinot/pull/17144#discussion_r2496374870
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2528,6 +2528,18 @@ public PauseState resumeTopicsConsumption(String tableNameWithType, List<Integer
}
private void sendForceCommitMessageToServers(String tableNameWithType, Set<String> consumingSegments) {
+ // For partial upsert tables, force-committing consuming segments is disabled.
+ // In some cases (especially when replication > 1), the server with fewer consumed rows
+ // was incorrectly chosen as the winner, causing other servers to reconsume rows
+ // and leading to inconsistent data.
+ // TODO: Temporarily disabled until a proper fix is implemented.
+ TableConfig tableConfig = _helixResourceManager.getTableConfig(tableNameWithType);
Review Comment:
Worth make this a helper method in TableConfigUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #17144:
URL: https://github.com/apache/pinot/pull/17144#discussion_r2496500455
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2528,6 +2528,18 @@ public PauseState resumeTopicsConsumption(String tableNameWithType, List<Integer
}
private void sendForceCommitMessageToServers(String tableNameWithType, Set<String> consumingSegments) {
+ // For partial upsert tables, force-committing consuming segments is disabled.
+ // In some cases (especially when replication > 1), the server with fewer consumed rows
+ // was incorrectly chosen as the winner, causing other servers to reconsume rows
+ // and leading to inconsistent data.
+ // TODO: Temporarily disabled until a proper fix is implemented.
+ TableConfig tableConfig = _helixResourceManager.getTableConfig(tableNameWithType);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Partial Upsert - Disable Reload on consuming segment and Force Commit on Partial Upsert Tables [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal merged PR #17144:
URL: https://github.com/apache/pinot/pull/17144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org