You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "krishan1390 (via GitHub)" <gi...@apache.org> on 2026/05/26 12:56:33 UTC
[PR] Add auto-complete flag to LineageEntry [pinot]
krishan1390 opened a new pull request, #18584:
URL: https://github.com/apache/pinot/pull/18584
## Summary
- Adds a new boolean `autoCompleteLineageEntry` to `LineageEntry`. Default `false` preserves the existing client-driven completion semantics (`endReplaceSegments`).
- When set to `true`, the entry signals that an observer (e.g. a controller-side periodic task) is permitted to mark the entry completed automatically once all `segmentsTo` are ONLINE in the external view.
- `SegmentLineage` serialization is updated to be tolerant on read (accepts both 4-tuple legacy and 5-tuple new format) and conservative on write (emits a 5th element only when the flag is `true`), so rolling upgrades and mixed-version controllers can co-exist.
- `PinotHelixResourceManager`'s lineage state-transition sites (`endReplaceSegments`, two branches of `revertReplaceSegments`) preserve the flag when constructing the updated entry.
- No consumer of the flag is added in this PR — a follow-up will introduce the observer that uses it.
## Test plan
- [x] Extended `SegmentLineageTest` to round-trip a `LineageEntry` with `autoCompleteLineageEntry=true` through `ZNRecord`.
- [x] Added a test that simulates a legacy 4-tuple `ZNRecord` written by an older controller and asserts it parses with the flag defaulting to `false`.
- [x] Asserted that two entries differing only in the new flag are not equal.
- [x] Confirmed default-false entries still serialize as 4-tuples (no wire-format churn for existing users).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add auto-complete flag to LineageEntry [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18584:
URL: https://github.com/apache/pinot/pull/18584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add auto-complete flag to LineageEntry [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18584:
URL: https://github.com/apache/pinot/pull/18584#discussion_r3330194112
##########
pinot-common/src/main/java/org/apache/pinot/common/lineage/SegmentLineage.java:
##########
@@ -171,7 +176,11 @@ public ZNRecord toZNRecord() {
String segmentsTo = String.join(COMMA_SEPARATOR, lineageEntry.getSegmentsTo());
String state = lineageEntry.getState().toString();
String timestamp = Long.toString(lineageEntry.getTimestamp());
- List<String> listEntry = Arrays.asList(segmentsFrom, segmentsTo, state, timestamp);
+ // Omit the 5th element when the flag is the default (false): keeps the wire format
+ // identical to legacy 4-tuples for every entry that does not opt into observer-driven
+ // completion, so older readers continue to parse them.
+ List<String> listEntry = lineageEntry.isAutoCompleteLineageEntry() ? Arrays.asList(segmentsFrom, segmentsTo,
Review Comment:
Writing a 5-tuple here breaks mixed-version controller compatibility. Older readers still expect exactly 4 list elements for a lineage entry, so the first `autoCompleteLineageEntry=true` write can make pre-upgrade controllers fail to parse the shared ZK record during rolling upgrade or rollback. Because this is controller-shared metadata, the new flag needs a wire-compatible encoding that old readers ignore, or a gated rollout plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add auto-complete flag to LineageEntry [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #18584:
URL: https://github.com/apache/pinot/pull/18584#discussion_r3346278841
##########
pinot-common/src/main/java/org/apache/pinot/common/lineage/SegmentLineage.java:
##########
@@ -171,7 +176,11 @@ public ZNRecord toZNRecord() {
String segmentsTo = String.join(COMMA_SEPARATOR, lineageEntry.getSegmentsTo());
String state = lineageEntry.getState().toString();
String timestamp = Long.toString(lineageEntry.getTimestamp());
- List<String> listEntry = Arrays.asList(segmentsFrom, segmentsTo, state, timestamp);
+ // Omit the 5th element when the flag is the default (false): keeps the wire format
+ // identical to legacy 4-tuples for every entry that does not opt into observer-driven
+ // completion, so older readers continue to parse them.
+ List<String> listEntry = lineageEntry.isAutoCompleteLineageEntry() ? Arrays.asList(segmentsFrom, segmentsTo,
Review Comment:
That is fine and expected. We will only enable features to use this flag after the rollout is successful and we don't rollback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add auto-complete flag to LineageEntry [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18584:
URL: https://github.com/apache/pinot/pull/18584#issuecomment-4544596124
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18584?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `69.23077%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.75%. Comparing base ([`0a8ac02`](https://app.codecov.io/gh/apache/pinot/commit/0a8ac02c8a062ced0f8542f5d0351b0ab2e31db4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c36f3aa`](https://app.codecov.io/gh/apache/pinot/commit/c36f3aae81046cc3ffa0e4e0919c4d2d7c3ee13c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18584?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../org/apache/pinot/common/lineage/LineageEntry.java](https://app.codecov.io/gh/apache/pinot/pull/18584?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Flineage%2FLineageEntry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9MaW5lYWdlRW50cnkuamF2YQ==) | 75.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/pinot/common/lineage/SegmentLineage.java](https://app.codecov.io/gh/apache/pinot/pull/18584?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Flineage%2FSegmentLineage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9TZWdtZW50TGluZWFnZS5qYXZh) | 60.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (0a8ac02) and HEAD (c36f3aa). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (0a8ac02) | HEAD (c36f3aa) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18584 +/- ##
============================================
- Coverage 64.27% 56.75% -7.52%
+ Complexity 1137 1 -1136
============================================
Files 3335 2564 -771
Lines 205898 148958 -56940
Branches 32129 24101 -8028
============================================
- Hits 132333 84543 -47790
+ Misses 62931 57225 -5706
+ Partials 10634 7190 -3444
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.75% <69.23%> (-7.52%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.75% <69.23%> (-7.52%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.75% <69.23%> (-7.52%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.75% <69.23%> (-0.01%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18584?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org