You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2025/08/07 18:59:48 UTC
[PR] [multistage] Push Collation from Exchange to Lite Mode Sort [pinot]
ankitsultana opened a new pull request, #16551:
URL: https://github.com/apache/pinot/pull/16551
### Summary
Minor improvement to pushdown collation from the enclosing Exchange to the Lite Mode generated Sort.
This is good for queries like the following. Without this patch the generated Sort only has a fetch and no collation.
```
SET useMultistageEngine = true;
SET usePhysicalOptimizer = true;
SET useLiteMode = true;
EXPLAIN PLAN FOR WITH ordered_events AS (
SELECT
cityName,
tripAmount,
ROW_NUMBER() OVER (
ORDER BY ts DESC
) as row_num
FROM userFactEvents
),
filtered_events AS (
SELECT
*
FROM ordered_events
WHERE row_num < 1000
)
SELECT
cityName,
SUM(tripAmount) as cityTotal
FROM filtered_events
GROUP BY cityName
```
The plan now for this query is:
```
Execution Plan
PhysicalAggregate(group=[{0}], agg#0=[$SUM0($1)], aggType=[DIRECT])
PhysicalFilter(condition=[<($3, 1000)])
PhysicalWindow(window#0=[window(order by [2 DESC] rows between UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])])
PhysicalExchange(exchangeStrategy=[SINGLETON_EXCHANGE], collation=[[2 DESC]])
PhysicalSort(fetch=[100000])
PhysicalProject(cityName=[$3], tripAmount=[$7], ts=[$9])
PhysicalTableScan(table=[[default, userFactEvents]])
```
### Test Plan
Added a UT and also verified on Quickstart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Push Collation from Exchange to Lite Mode Sort [pinot]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #16551:
URL: https://github.com/apache/pinot/pull/16551#issuecomment-3165988941
I was wondering which phase generate the PhysicalSort(fetch=[10000]) without the collation before this PR. IIUC, the Calcite will generate the LogicalWindow Node with collocation and doesn't create a LogicalSort Node along the way. If so, it means the pinot has glitches when translate from LogicalWindow to Physical Window by wrongly inserting the PhysicalSort Node without collation.
Moreover, what's the behavior of the general MSE plan? Shall we apply the same patch for non lite mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Push Collation from Exchange to Lite Mode Sort [pinot]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #16551:
URL: https://github.com/apache/pinot/pull/16551#issuecomment-3169562360
> I was wondering which phase generate the PhysicalSort(fetch=[10000]) without the collation before this PR. IIUC, the Calcite will generate the LogicalWindow Node with collocation and doesn't create a LogicalSort Node along the way. If so, it means the pinot has glitches when translate from LogicalWindow to Physical Window by wrongly inserting the PhysicalSort Node without collation.
>
> Moreover, what's the behavior of the general MSE plan? Shall we apply the same patch for non lite mode?
Ankit provides the answers in slack. I pasted the explanation for future reference:
>
The default MSE generates a SortExchange under the LogicalWindow WindowExchangeNodeInsertRule if a collation is applicable.
Physical optimizer uses TraitAssignment and adds a trait under the Window for the collation. [See](https://github.com/apache/pinot/blob/584119fce06d47387e8b68a9eceb8493eebe7b02/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/traits/TraitAssignment.java#L192)
For Lite Mode, we add a sort in the leaf stage. Currently the sort node was blindly added with a row limit. But, it could be that the Exchange just above leaf stage had a collation. That would happen when the node above Exchange had pushed down a Collation trait. So this bug was kinda specific to lite mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Push Collation from Exchange to Lite Mode Sort [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16551:
URL: https://github.com/apache/pinot/pull/16551#issuecomment-3165517527
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16551?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `75.00000%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.32%. Comparing base ([`9f74fc7`](https://app.codecov.io/gh/apache/pinot/commit/9f74fc7d14808e85515ae4ddf906f2d805b3a714?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a74ef0d`](https://app.codecov.io/gh/apache/pinot/commit/a74ef0df1fc6bf055c374b83ec265aa7d8df63df?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16551?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../physical/v2/opt/rules/LiteModeSortInsertRule.java](https://app.codecov.io/gh/apache/pinot/pull/16551?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fphysical%2Fv2%2Fopt%2Frules%2FLiteModeSortInsertRule.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9waHlzaWNhbC92Mi9vcHQvcnVsZXMvTGl0ZU1vZGVTb3J0SW5zZXJ0UnVsZS5qYXZh) | 75.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (9f74fc7) and HEAD (a74ef0d). Click for more details.
>
> <details><summary>HEAD has 36 uploads less than BASE</summary>
>
>| Flag | BASE (9f74fc7) | HEAD (a74ef0d) |
>|------|------|------|
>|unittests2|2|0|
>|unittests|4|1|
>|java-21|5|1|
>|temurin|10|1|
>|integration|6|0|
>|integration1|2|0|
>|java-11|5|0|
>|custom-integration1|2|0|
>|integration2|2|0|
>|unittests1|2|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #16551 +/- ##
============================================
- Coverage 63.25% 56.32% -6.93%
+ Complexity 1362 699 -663
============================================
Files 3012 2356 -656
Lines 174294 129605 -44689
Branches 26684 20606 -6078
============================================
- Hits 110244 73006 -37238
+ Misses 55644 50574 -5070
+ Partials 8406 6025 -2381
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.32% <75.00%> (-6.91%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.32% <75.00%> (-6.93%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.32% <75.00%> (-6.92%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.32% <75.00%> (-0.09%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16551/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16551?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [multistage] Push Collation from Exchange to Lite Mode Sort [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana merged PR #16551:
URL: https://github.com/apache/pinot/pull/16551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org