You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "yashmayya (via GitHub)" <gi...@apache.org> on 2026/07/15 23:16:49 UTC
[PR] [multistage] Add tests for chaining colocated operations without data shuffle [pinot]
yashmayya opened a new pull request, #18997:
URL: https://github.com/apache/pinot/pull/18997
Follow-up to #18804. Adds test coverage verifying that **colocated operations can be chained without a data shuffle**: when a chain of joins / window functions / set operations is all keyed on the same partition column and the colocation hints are applied, every inter-stage hash exchange in the plan is pre-partitioned (direct, 1-to-1) and no full shuffle is inserted anywhere in the chain.
This exercises the interaction of the three colocation hints:
- `joinOptions(is_colocated_by_join_keys='true')`
- `windowOptions(is_partitioned_by_window_keys='true')`
- `setOpOptions(is_colocated_by_set_op_keys='true')`
**Tests added (test-only, no production changes):**
- `QueryCompilationTest`: a planner-level invariant check (`testColocatedOperationsChainWithoutShuffle`) that asserts every hash-distributed exchange in the plan is pre-partitioned, across four chains:
- colocated join → window
- set op of two colocated joins
- set op of two colocated windows
- the triple: colocated join → window → set op
Plus a baseline (`testChainWithoutColocationHintsStillShuffles`) confirming the same chain *does* shuffle without the hints, so the assertions are not vacuous.
- `QueryHints.json`: two H2-validated runtime cases (a colocated join → window, and a `UNION ALL` of two colocated joins) on tables physically partitioned by the key, confirming the chained colocation also produces correct results on both the V1 and V2 engines.
**Hint placement note:** because a set operation is an ancestor of its branch `SELECT`s, chaining through a set op uses the outer-`SELECT`-wrap form for the set-op hint with the join/window hints on each branch — this is what lets colocation propagate through every level of the chain.
**Note:** this PR is stacked on #18804 (which introduces the `setOpOptions` hint the set-op chains depend on). Its diff includes that commit until #18804 merges — the commit to review here is the top one (`Add tests for chaining colocated joins, window functions and set operations without shuffle`); I'll rebase once #18804 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Add tests for chaining colocated operations without data shuffle [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya merged PR #18997:
URL: https://github.com/apache/pinot/pull/18997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Add tests for chaining colocated operations without data shuffle [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18997:
URL: https://github.com/apache/pinot/pull/18997#issuecomment-4986614526
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18997?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `5.88235%` with `16 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 37.71%. Comparing base ([`fd772ee`](https://app.codecov.io/gh/apache/pinot/commit/fd772eedd5fb849f6aee5067f3509c768a326ece?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`069f4c0`](https://app.codecov.io/gh/apache/pinot/commit/069f4c036c69b3221124103de377d2636b121f9d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18997?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...te/rel/rules/PinotSetOpExchangeNodeInsertRule.java](https://app.codecov.io/gh/apache/pinot/pull/18997?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Frules%2FPinotSetOpExchangeNodeInsertRule.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvcnVsZXMvUGlub3RTZXRPcEV4Y2hhbmdlTm9kZUluc2VydFJ1bGUuamF2YQ==) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18997?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/pinot/calcite/rel/hint/PinotHintOptions.java](https://app.codecov.io/gh/apache/pinot/pull/18997?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcalcite%2Frel%2Fhint%2FPinotHintOptions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2FsY2l0ZS9yZWwvaGludC9QaW5vdEhpbnRPcHRpb25zLmphdmE=) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18997?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (fd772ee) and HEAD (069f4c0). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (fd772ee) | HEAD (069f4c0) |
>|------|------|------|
>|java-21|5|1|
>|unittests1|1|0|
>|unittests|2|1|
>|temurin|5|1|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18997 +/- ##
=============================================
- Coverage 65.21% 37.71% -27.50%
+ Complexity 1405 1398 -7
=============================================
Files 3418 3418
Lines 214651 214667 +16
Branches 33922 33927 +5
=============================================
- Hits 139989 80967 -59022
- Misses 63356 126290 +62934
+ Partials 11306 7410 -3896
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.71% <5.88%> (-27.50%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.71% <5.88%> (-27.50%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.71% <5.88%> (-27.50%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18997/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.71% <5.88%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/18997?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org