You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/02/12 01:15:33 UTC
[PR] feat: sort file groups by statistics during sort pushdown [datafusion]
adriangb opened a new pull request, #20304:
URL: https://github.com/apache/datafusion/pull/20304
## Summary
- Sort files within each file group by min/max statistics during sort pushdown to better align with the requested ordering
- When files are non-overlapping and within-file ordering is guaranteed (Parquet returns `Exact`), the `SortExec` is completely eliminated
- When files overlap, best-effort statistics-based reordering is applied with `SortExec` retained for correctness (`Inexact`)
- `ParquetSource::try_pushdown_sort` now returns `Exact` when the file's natural ordering already satisfies the request, enabling sort elimination for the same-direction case
- Add SLT integration tests covering non-overlapping sort elimination, overlapping files, reverse scan with mixed file naming, and multi-group merging
Related to https://github.com/apache/datafusion/issues/19724
## Test plan
- [x] `cargo test -p datafusion-sqllogictest --test sqllogictests -- sort_pushdown` passes
- [x] `cargo test -p datafusion-sqllogictest --test sqllogictests -- parquet_sorted_statistics` passes
- [x] New SLT tests verify EXPLAIN plans show correct optimizer behavior (sort elimination, SortExec retention, reverse_row_groups, file ordering)
- [x] New SLT tests verify query result correctness for all scenarios
๐ค Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: sort file groups by statistics during sort pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20304:
URL: https://github.com/apache/datafusion/pull/20304#issuecomment-4010507999
๐ค `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing sort-groups (29ee72eb65aa920f91b8d510e15f92ca82182bb3) to a5f490e4c31125b8126105385c6c1120cb2fa2de [diff](https://github.com/apache/datafusion/compare/a5f490e4c31125b8126105385c6c1120cb2fa2de..29ee72eb65aa920f91b8d510e15f92ca82182bb3) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: sort file groups by statistics during sort pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20304:
URL: https://github.com/apache/datafusion/pull/20304#issuecomment-3893723058
I'm going to wait to sequence this after #20329. I think this also fixes the bug but I'd rather do bugfix and feature independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: sort file groups by statistics during sort pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed pull request #20304: feat: sort file groups by statistics during sort pushdown
URL: https://github.com/apache/datafusion/pull/20304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: sort file groups by statistics during sort pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20304:
URL: https://github.com/apache/datafusion/pull/20304#issuecomment-4010359790
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: sort file groups by statistics during sort pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20304:
URL: https://github.com/apache/datafusion/pull/20304#issuecomment-4010676751
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and sort-groups
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ sort-groups โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.57 ms โ 2.64 ms โ no change โ
โ QQuery 1 โ 49.81 ms โ 51.68 ms โ no change โ
โ QQuery 2 โ 163.21 ms โ 170.26 ms โ no change โ
โ QQuery 3 โ 164.68 ms โ 170.01 ms โ no change โ
โ QQuery 4 โ 1118.70 ms โ 1176.20 ms โ 1.05x slower โ
โ QQuery 5 โ 1304.76 ms โ 1420.54 ms โ 1.09x slower โ
โ QQuery 6 โ 6.61 ms โ 6.80 ms โ no change โ
โ QQuery 7 โ 55.17 ms โ 55.77 ms โ no change โ
โ QQuery 8 โ 1571.66 ms โ 1604.62 ms โ no change โ
โ QQuery 9 โ 1958.40 ms โ 2008.11 ms โ no change โ
โ QQuery 10 โ 341.86 ms โ 361.48 ms โ 1.06x slower โ
โ QQuery 11 โ 397.43 ms โ 416.61 ms โ no change โ
โ QQuery 12 โ 1243.33 ms โ 1307.39 ms โ 1.05x slower โ
โ QQuery 13 โ 2051.70 ms โ 2126.89 ms โ no change โ
โ QQuery 14 โ 1243.47 ms โ 1305.09 ms โ no change โ
โ QQuery 15 โ 1280.20 ms โ 1336.19 ms โ no change โ
โ QQuery 16 โ 2690.15 ms โ 2789.21 ms โ no change โ
โ QQuery 17 โ 2667.07 ms โ 2732.10 ms โ no change โ
โ QQuery 18 โ 5484.33 ms โ 5328.09 ms โ no change โ
โ QQuery 19 โ 130.29 ms โ 131.86 ms โ no change โ
โ QQuery 20 โ 1874.78 ms โ 1914.24 ms โ no change โ
โ QQuery 21 โ 2157.82 ms โ 2310.22 ms โ 1.07x slower โ
โ QQuery 22 โ 3721.26 ms โ 3963.04 ms โ 1.06x slower โ
โ QQuery 23 โ 11896.30 ms โ 12766.66 ms โ 1.07x slower โ
โ QQuery 24 โ 203.36 ms โ 215.42 ms โ 1.06x slower โ
โ QQuery 25 โ 421.89 ms โ 465.67 ms โ 1.10x slower โ
โ QQuery 26 โ 202.41 ms โ 199.01 ms โ no change โ
โ QQuery 27 โ 2722.68 ms โ 2720.60 ms โ no change โ
โ QQuery 28 โ 25510.30 ms โ 25425.30 ms โ no change โ
โ QQuery 29 โ 1061.23 ms โ 1081.94 ms โ no change โ
โ QQuery 30 โ 1277.06 ms โ 1328.88 ms โ no change โ
โ QQuery 31 โ 1385.31 ms โ 1425.15 ms โ no change โ
โ QQuery 32 โ 5106.58 ms โ 4828.95 ms โ +1.06x faster โ
โ QQuery 33 โ 5655.99 ms โ 5886.97 ms โ no change โ
โ QQuery 34 โ 6464.05 ms โ 6298.75 ms โ no change โ
โ QQuery 35 โ 2069.78 ms โ 2076.51 ms โ no change โ
โ QQuery 36 โ 186.71 ms โ 193.12 ms โ no change โ
โ QQuery 37 โ 71.32 ms โ 75.21 ms โ 1.05x slower โ
โ QQuery 38 โ 109.19 ms โ 114.37 ms โ no change โ
โ QQuery 39 โ 329.77 ms โ 344.86 ms โ no change โ
โ QQuery 40 โ 43.15 ms โ 38.72 ms โ +1.11x faster โ
โ QQuery 41 โ 37.98 ms โ 35.94 ms โ +1.06x faster โ
โ QQuery 42 โ 32.57 ms โ 30.90 ms โ +1.05x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 96466.87ms โ
โ Total Time (sort-groups) โ 98241.97ms โ
โ Average Time (HEAD) โ 2243.42ms โ
โ Average Time (sort-groups) โ 2284.70ms โ
โ Queries Faster โ 4 โ
โ Queries Slower โ 10 โ
โ Queries with No Change โ 29 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org