You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2026/09/05 21:03:23 UTC
[PR] [query-runtime] Add adaptive partition coalescing policy [pinot]
wirybeaver opened a new pull request, #19470:
URL: https://github.com/apache/pinot/pull/19470
## Summary
This PR adds the smallest safe foundation for adaptive query execution (AQE) in Pinot's multi-stage engine: a pure partition-coalescing policy driven by completed producer partition sizes.
Ballista can re-plan an undispatched consumer after a materialized producer stage completes. Pinot currently dispatches an eager streaming mailbox graph whose endpoints already encode worker assignments, so changing stage parallelism after dispatch would be unsafe. This change therefore implements only the reusable decision seam and does not mutate a running query.
## Design
```text
Ballista AQE
+--------------------+ +----------------+ +--------------------+
| completed producer | --> | shuffle stats | --> | lazy consumer |
| materialized data | | and replanning | | stage dispatch |
+--------------------+ +----------------+ +--------------------+
Pinot MSE today
+----------------+ eager whole-graph dispatch +------------------+
| physical plan | ---------------------------------> | streaming mailbox|
+----------------+ | topology |
+------------------+
|
immutable after dispatch
X
runtime stage resizing
Safe first Pinot AQE seam
ordered completed-partition byte sizes + target size
|
v
+------------------------------+
| AdaptivePartitionCoalescer |
| deterministic contiguous |
| immutable partition ranges |
+------------------------------+
|
v
future undispatched consumer-stage assignment
```
`AdaptivePartitionCoalescer` walks partitions from left to right and:
- coalesces adjacent small groups using Ballista's production thresholds;
- preserves source order and assigns each partition exactly once;
- keeps oversized source partitions indivisible;
- returns the original assignment when parallelism cannot decrease;
- uses exact intermediate byte totals before threshold comparison;
- returns immutable half-open partition ranges.
The policy is intentionally independent of mailbox, RPC, planner, worker-assignment, and live-dispatch classes. Applying it to queries remains deferred until Pinot supports staged dispatch and materialized or late-bound exchange data.
## Test Plan
- `./mvnw -pl pinot-query-runtime -am -Dtest=AdaptivePartitionCoalescerTest -Dsurefire.failIfNoSpecifiedTests=false test`
- `./mvnw spotless:apply -pl pinot-query-runtime`
- `./mvnw checkstyle:check -pl pinot-query-runtime`
- `./mvnw license:format -pl pinot-query-runtime`
- `./mvnw license:check -pl pinot-query-runtime`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [query-runtime] Add adaptive partition coalescing policy [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19470:
URL: https://github.com/apache/pinot/pull/19470#issuecomment-5554962779
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19470?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `93.87755%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.81%. Comparing base ([`e6787e1`](https://app.codecov.io/gh/apache/pinot/commit/e6787e178645fd3c5a887e2aba7e0de8343cd524?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`21ad1af`](https://app.codecov.io/gh/apache/pinot/commit/21ad1afaa32c491b516f29e6dc7b4141e0afb4fa?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19470?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...y/service/dispatch/AdaptivePartitionCoalescer.java](https://app.codecov.io/gh/apache/pinot/pull/19470?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fdispatch%2FAdaptivePartitionCoalescer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9BZGFwdGl2ZVBhcnRpdGlvbkNvYWxlc2Nlci5qYXZh) | 93.87% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e6787e1) and HEAD (21ad1af). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (e6787e1) | HEAD (21ad1af) |
>|------|------|------|
>|unittests|2|1|
>|java-25|6|1|
>|temurin|6|1|
>|lane-a|2|0|
>|integration|4|0|
>|integration1|2|0|
>|lane-b|2|0|
>|integration2|2|0|
>|unittests2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19470 +/- ##
============================================
- Coverage 67.69% 57.81% -9.89%
+ Complexity 1430 1 -1429
============================================
Files 3488 2686 -802
Lines 224306 164136 -60170
Branches 35416 26667 -8749
============================================
- Hits 151850 94895 -56955
- Misses 60447 61212 +765
+ Partials 12009 8029 -3980
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.81% <93.87%> (-9.89%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.81% <93.87%> (-9.89%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.81% <93.87%> (-9.89%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.81% <93.87%> (+0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19470?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org