You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/06/10 22:51:20 UTC
[PR] Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere [pinot]
xiangfu0 opened a new pull request, #18731:
URL: https://github.com/apache/pinot/pull/18731
## Problem
In strict replica-group routing (`StrictReplicaGroupInstanceSelector`, and `ReplicaGroupInstanceSelector` for upsert/dedup tables), when an old segment is missing (not ONLINE/CONSUMING in ExternalView) on an instance, that instance is marked unavailable **for every segment sharing the same ideal state instance set**. This is correct when the segment is available on some other instance in the group: it forces the whole group to route to an instance that has all the segments.
However, when a segment is not online on **any** instance in its instance set, this logic marks **all** instances of the group unavailable, taking down every other segment hosted by the same instances — even though excluding instances cannot make the missing segment servable and provides no consistency benefit.
This caused a production incident: during tier relocation while one cold-tier replica group was restarting, a `bestEfforts` rebalance advanced the IdealState past ExternalView convergence, leaving a handful of relocating segments ONLINE nowhere for ~1 minute. Those few segments poisoned both cold instances of every instance group, and the broker reported the **entire cold tier (~29.5k segments) unavailable**:
```
29562 segments unavailable, sampling 10: [...], with routing policy: strictReplicaGroup [realtime]
```
The same mechanism also lets a single corrupt segment (ERROR on all replicas) black out a whole replica-group set.
## Fix
In `updateSegmentMapsForUpsertTable`, an old segment with no online instance within its ideal state instance set no longer contributes to the unavailable-instance computation for its group. It gets an empty candidate list and is reported as unavailable **individually**, while all other segments on those instances keep serving.
The strict replica-group guarantee is unchanged for any segment that is available on at least one instance: instances missing such a segment are still excluded for the whole group, so all served segments of a group are still served from a consistent instance (verified by the last phase of the new regression test).
## Behavior change
Before: a segment unavailable everywhere caused **all** segments sharing its instance set to be reported unavailable, and the query served no rows from any of them.
After: only the truly unservable segment is reported unavailable; the rest of the group keeps serving.
This applies to `StrictReplicaGroupInstanceSelector` (all tables) and `ReplicaGroupInstanceSelector` (upsert/dedup tables), since both share `updateSegmentMapsForUpsertTable`. For upsert tables this means a query can now return rows from the remaining segments of a partition while the nowhere-online segment is reported unavailable; records whose latest version lives in that segment may surface earlier versions. Note that in both the old and the new behavior the query response carries a `BROKER_SEGMENT_UNAVAILABLE` exception, so clients that treat partial results as failures are unaffected. Given that the previous behavior also returned partial (more incomplete) results with the same exception, this is shipped without a config gate.
## Testing
- New regression test `testStrictReplicaGroupSegmentNotOnlineAnywhereDoesNotMarkGroupDown` covering: (1) the incident scenario (relocated segment not online anywhere does not take down the group), (2) recovery once the segment loads, and (3) the preserved group-down behavior for a segment that is online on one instance but missing on another.
- Updated existing test expectations that codified the old blackout behavior (each with explanatory comments).
- Full `pinot-broker` module test suite passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18731:
URL: https://github.com/apache/pinot/pull/18731#discussion_r3395695007
##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/ReplicaGroupInstanceSelector.java:
##########
@@ -222,10 +227,28 @@ void updateSegmentMapsForUpsertTable(IdealState idealState, ExternalView externa
// Calculate the unavailable instances based on the old segments' online instances for each combination of instances
// in the ideal state
Map<Set<String>, Set<String>> unavailableInstancesMap = new HashMap<>();
+ List<String> segmentsNotOnlineAnywhere = null;
for (Map.Entry<String, Set<String>> entry : oldSegmentToOnlineInstancesMap.entrySet()) {
String segment = entry.getKey();
Set<String> onlineInstances = entry.getValue();
Map<String, String> idealStateInstanceStateMap = idealStateAssignment.get(segment);
+ // When a segment is not online on any instance in the ideal state (e.g. it was just moved to a new set of
+ // instances by tier relocation/rebalance and is still being loaded on all of them), no instance selection can
+ // serve it, so excluding instances for the whole group cannot help consistency. Marking all its instances
+ // unavailable would instead take down every other segment sharing the same instances (a production incident
+ // showed a whole tier of ~30k segments reported unavailable due to a few relocating segments). Skip it here;
+ // it gets an empty candidate list below and is reported unavailable individually.
+ if (onlineInstances.isEmpty()) {
Review Comment:
This shared helper is also the path that `ReplicaGroupInstanceSelector.updateSegmentMaps()` uses for upsert/dedup tables. Skipping the group-wide exclusion here changes those tables from "partition unavailable" to "serve older rows": if the latest segment for a PK is the nowhere-online segment, we now route to an older segment in the same group and surface a superseded value. That is a correctness regression, not just an availability tradeoff. This exception likely needs to stay out of the upsert/dedup path (or be gated separately) instead of applying in the shared helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #18731: Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere
URL: https://github.com/apache/pinot/pull/18731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18731:
URL: https://github.com/apache/pinot/pull/18731#issuecomment-4675759560
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18731?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 57.02%. Comparing base ([`b0cb24b`](https://app.codecov.io/gh/apache/pinot/commit/b0cb24b0e89a352b0ad48ffa637fbf27504fc6bc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`91a8e37`](https://app.codecov.io/gh/apache/pinot/commit/91a8e37a8b2283c0a9c3d9295158f4185f376573?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (b0cb24b) and HEAD (91a8e37). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (b0cb24b) | HEAD (91a8e37) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18731 +/- ##
============================================
- Coverage 64.59% 57.02% -7.58%
+ Complexity 1305 1 -1304
============================================
Files 3373 2584 -789
Lines 208894 150351 -58543
Branches 32667 24335 -8332
============================================
- Hits 134937 85735 -49202
+ Misses 63116 57345 -5771
+ Partials 10841 7271 -3570
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.02% <ø> (-7.58%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.02% <ø> (-7.58%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.02% <ø> (-7.57%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.02% <ø> (-0.02%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18731/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/18731?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix strictReplicaGroup routing to not mark the whole instance group unavailable for segments not online anywhere [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18731:
URL: https://github.com/apache/pinot/pull/18731#issuecomment-4685296255
Closing this in favor of fixing the root cause in TableRebalancer: the broker-side change only caps the blast radius after segments have already become unservable, and for upsert tables serving the remaining segments of a partition could surface stale values, which is arguably worse than failing loudly. The real issue is that a best-efforts rebalance can advance the IdealState past ExternalView convergence and drop the last actually-serving replica of a segment, leaving it ONLINE nowhere. Will follow up with a TableRebalancer fix that makes the min-available-replicas check ExternalView-aware so this state can never be created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org