You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/05/14 07:12:26 UTC
[PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip I/O [datafusion-site]
zhuqi-lucas opened a new pull request, #186:
URL: https://github.com/apache/datafusion-site/pull/186
## Summary
A walkthrough of the sort pushdown work landed and in flight on Apache DataFusion. Opening as a **draft** to share the narrative early — the in-flight PRs the post discusses are still in flight, but the structure and the merged work (Phase 1 #19064, Phase 2 #21182, BufferExec #21426, row-group reverse #18817) are in their final shape.
## What this post covers
- Why `SortExec` is expensive, and what `Exact` / `Inexact` mean at *runtime* (static `fetch` vs `TopK` dynamic filter).
- **Phase 1** ([#19064](https://github.com/apache/datafusion/pull/19064)) — the `PushdownSort` rule + reverse row-group case.
- **Phase 2** ([#21182](https://github.com/apache/datafusion/pull/21182)) — statistics-based file sort that upgrades `Unsupported` to `Exact`, eliminating the `SortExec` on non-overlapping ASC scans. Includes the `BufferExec` compensation ([#21426](https://github.com/apache/datafusion/pull/21426)) so the SPM above doesn't lose its implicit memory buffer.
- **Reverse scans** — today's row-group reverse ([#18817](https://github.com/apache/datafusion/pull/18817), Inexact) and the community decision to wait for arrow-rs page-level reverse ([apache/arrow-rs#9937](https://github.com/apache/arrow-rs/pull/9937)) before pursuing `Exact` reverse, after memory-profile pushback on the original row-group-level proposal.
- **Benchmarks** — 2.1×–49× on the ASC-LIMIT `sort_pushdown` suite.
- **What's next** — the dynamic / TopK-driven path (#21351 merged, #21733, #21712, #21956, #21580) including the precise RG-level pruning vs mid-stream early return distinction, and the `EnsureRequirements` unification ([#21976](https://github.com/apache/datafusion/pull/21976)).
- Links into the prior [dynamic filters](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/) and [limit pruning](https://datafusion.apache.org/blog/2026/03/20/limit-pruning/) posts so the series reads as a coherent thread.
## Why a draft
A few of the in-flight PRs the "What's next" section references may evolve in review (e.g. #21580 may be split into smaller pieces, dynamic RG scheduling on top of #21351 is described but not yet on a PR). Opening as draft so we can adjust wording as those land or change shape — happy to flip to ready for review when the dust settles, or earlier if reviewers prefer.
## Test plan
- [x] Rendered locally with the Pelican Docker image from the project README — images, internal links, code blocks, and tables all render correctly.
- [x] All issue / PR / blog-post links checked against current state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4979810990
I hope to complete my pass by the end of this week. Thank you for your patience
i think this is an important post so that is why I have been trying to spend more time on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5013889473
Thanks a lot for the deep pass @alamb — this reads much better than what I put up. Structure changes (buffering + decoder-loop moved to appendices, Related Work at the end, cleaner Exact/Inexact split) are all improvements.
Pushed one small commit (1666952) with three narrow tweaks:
- **Row-Level Early Stopping paragraph**: reworded so `#22450`'s contribution is described as "keeps the threshold up-to-date at row-group boundaries" rather than "adds projection-column skip". The projection-column skip itself lives in arrow-rs `RowFilter` — `#22450` just makes it fire more often.
- **Conclusion**: replaced "In DataFusion 54, we implemented …" with "Over the last few DataFusion releases we implemented …". `#22450` merged 2026-06-22, after `54.0.0` was tagged, so it will ship in a later release.
- **Decision-tree figure alt-text**: `try_pushdown_sort` → `PushdownSort` to match the struct name used throughout the surrounding prose.
Happy to add per-query `sort_tpch` numbers as you suggested — I have them locally, will add in a follow-up. And +1 on planning publish for Monday July 20th.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544648131
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
Review Comment:
Done — added the transition sentence.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
Review Comment:
Done — dropped `across a shared work-stealing queue`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
Review Comment:
Done — dropped the reverse-satisfies parenthetical.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
Review Comment:
Done — renamed to `File-level early stop`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544644957
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
Review Comment:
Ack — kept `SortExec` here as this section is DataFusion internals.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
Review Comment:
Done — applied suggested phrasing.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
Review Comment:
Done — dropped the `runs late after EnsureRequirements` and plan-walk detail.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
Review Comment:
Done — adopted suggestion verbatim.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
Review Comment:
Done — removed the two conservative bail-outs paragraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4915752207
@alamb — all 46 comments from your round-2 review addressed in 4daaa2d + 9b48aa8:
- 44 text changes applied (concise per-comment ✅ replies inline).
- 2 figure-annotation comments (transition_anatomy / pruner_loop): the figures had overflow issues and their captions were flagged as belonging in the blog text; since the surrounding markdown already describes drain/decide/drive and watch/rebuild/prune, dropping the figures resolved both concerns without losing content.
Ready for another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4938328145
taking a final pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4878523976
Starting to check this one out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4915514245
Thanks a lot @alamb for great suggestion and review! I am addressing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544701037
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — removed the `transition_anatomy.png` figure; the surrounding markdown text already covers the drain/decide/drive loop, so no overflow issue remains and the figure captions aren't needed.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — removed the `pruner_loop.png` figure; the surrounding markdown text already covers watch/rebuild/prune, so no overflow issue remains and the figure captions aren't needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544053628
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
Review Comment:
I think you can make this more concise and thus understandable by removing some of the detail and the bullets
Something like
> Stats-based sort elimination handles the case when the table has a
> declared `output_ordering` *and* the files are provably
> non-overlapping after sorting by min. However when these cases
> do not apply, a full sort is still overkill. The parquet metadata ...
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
Review Comment:
I recomment removing this heading as it is covered by the text below
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
Review Comment:
I found this wording slight confusing -- the "leaf source" of what? Maybe it would be clearer if it were something like
"for each `SortExec` that `SortPushdown` tries to push down into the scan, the data source returns `Exact`, ...."
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
Review Comment:
This section might be easier to understand with a bit more of a transition:
> If SortPushdown determines `Inexact` applies to the data source, and the
> source supports `Inexact` further optimizations are possible. For example, the
> parquet opener applies ....
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
Review Comment:
this image says "Part 4" which is a little confusing as well as also referring to #22450 which will not make sense to the average reader I don't think
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
Review Comment:
I rcommend calling this section "Row Group Filter early stop" for consistency with the section above and remove the link to #22450 (put that in the text)
THe reason I think this is more readable is that it removes a level of indirection -- as written readers now need to keep the "#22450" concept in their head and mentally translate it it to row group level pruning. Avoiding that translation step makes the content easier to understand
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
Review Comment:
This paragraph could also be rephrased as the introduction of the next section (and again I would avoid using a PR number to name the concept)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
Review Comment:
"Runtime Row-Group Dynamic Pruning"
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
Review Comment:
You can probably tighten this up / make it more concise
```suggestion
The pruner is designed so the expensive work only fires when it can
possibly help: a cheap epoch check tells it whether the dynamic filter
has actually changed since last time, and only then does it rebuild
the pruning predicate. The predicate is then applied to remaining
row groups' min/max statistics, using only metadata, no I/O.
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
Review Comment:
As the point of this blog is to explain these sort based optimizations, I recommend comparing:
1. Datafusion `54` without the sort optimizations (disable them)
2. DataFusion `54` with the sort optimizations
(This might be what it is doing, but the text says it is comparing the result of some particular PR)
That way the results are showing how effecive this approach is (rather than a particular PR)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Sme comments: <img width="926" height="434" alt="Image" src="https://github.com/user-attachments/assets/c252125e-af2d-4a5b-95f9-abda60222e5d" />
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
Review Comment:
Is there a reason this list uses inconsistent labeling (`0`, `A`, and `B` rather than `1`, `2`, `3`, or `A`, `B`, `C`?
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
Review Comment:
I think you can remove "(and for the reverse-satisfies-the-request cases above)." and not lose any of the main idea
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
Review Comment:
the "shared workstealing queue" doesn't seem relevant to how files are reordered and it just make it harder to understand this sentence -- I recommend removing it
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
Review Comment:
it might help here to name explcitly the `low-value` files to help readers understand the figure -- "low-value files (`file_d` and `file_c`). Also maybe it is worth explicitly pointing out that "low-value" means "sort key values are smaller for ASC wueries"
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
Review Comment:
I recommend removing the "already works" here -- so this blog is mostly about the current DataFusion capabilities, rather than the order of what was added
So like
### File-level early stop
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
Review Comment:
this only applies when filter pushdown is on, right? I think you could simplify this section by escribing this at a high level rather than details
Something like
> by reordering row groups so the ones most likely to contain the minimum value are read first
> makes the exsiting TopK dynamic filters + filter_pushdown optimization (link to the
> late materialization arrow blog) more effective, and they can often rule out entire row group
> after evaluating the filter columns, and avoid even fetch projection columns.
> This optimization was added in #22450
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
Review Comment:
Rather than focus on #22450 I think it would be clearer to read if this sentence was framed more like "DataFusion now also re-checks the dynamic filter **at every row-group boundary** ....
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
Review Comment:
I am not sure that describing the evolution of the internal control loop implementation is important for the main point of the article -- it will take mental energy to understand the history / evolution; I think it would be easier to read if you just summarized the current state of the decoder loop (aka the "loop and decision point" section)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
Review Comment:
It might also be good to point out this only happens if the dynamic filter has been updated since the initial check
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Some comments on the figure:
<img width="978" height="507" alt="Image" src="https://github.com/user-attachments/assets/2545e195-3931-43b4-973b-579117f7c3c2" />
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
Review Comment:
I found this section somewhat confusing too as it seems to be illustrating the multi-level pruning strategy, but the words are descrobing the RowFilter level pruning
I would suggest moving this section earlier in the blog to introduce the concept of multi-layer pruning (which is what the sort based optimizations really take advantage of)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
Review Comment:
I found this section confusing --
Some thoughts:
1. Call it "Example of pruning row groups using top k dynamic predicates"
2. Add an introductory sentence / parapgrah that explain what the section -- namely an example of how the techniques described in this blog work together to prune row groups.
It might be too much for the blog, but I think the example would be clearer if there were a second figure that shows how Row Group Pruning works without the dynamic row group reordering -- that way the benefit of the approach in this blog might be easier to understand in comparsion to another fiure
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
Review Comment:
This is confusing to me -- rather than say what the take away is NOT, how about just say what the result is -- several common timeseries workloads are accelerated 3-4x?
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
Review Comment:
If you want to explain the results like this I suggest moving the details of the tpch_sort benchmark to the intro
> The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
> shape `ORDER BY ... LIMIT 100`. The data is sorted by l_orderkey, stored in parquet, with each row group XXX size , 1.5 distinct values,....
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
+
+Two complementary directions are open. The first needs an upstream
+arrow-rs primitive; the second is pure DataFusion plumbing on top
+of [#22450]:
+
+### A · Page-level `Exact` reverse · arrow-rs [#9937]
+
+<img src="/blog/images/sort-pushdown/reverse-scan.svg" alt="Row-group reverse (128 MB peak) vs page-level reverse (1 MB peak)" width="100%" class="img-fluid"/>
+
+[#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+Today's `DESC` query support lives in the `Inexact` path: the
+row-group reverse emits "RGs descending × rows ascending", which is
+close to DESC but not strictly so. `SortExec` stays.
+
+A page-level reverse primitive in arrow-rs would let the reader
+walk the parquet offset index in reverse — decoding each page
+forward, reversing its `RecordBatch`, and emitting before moving to
+the previous page. Peak buffer drops from ~128 MB (one
+row group) to ~1 MB (one page); per-page decode stays forward (RLE,
+dictionary, delta, and bit-packed encodings are all forward-only
+by construction — page *traversal* is what gets reversed). Once
+each batch already comes out in DESC order, `PushdownSort` can
+finally return `Exact` for `DESC`, the `SortExec` is removed
+outright, and `LIMIT N` becomes a static fetch.
+
+In flight upstream as [arrow-rs#9937]. The killer use case is
+**filtered reverse `TopK`** — e.g. `WHERE user_id = 42 ORDER BY ts
+DESC LIMIT 10`. You can't pre-compute a `RowSelection::with_limit`
+because matching rows are sparse; the only correct strategy is to
+stream pages backward, filter, and stop when 10 matches accumulate.
+Row-group reverse stops at ~128 MB granularity; page reverse stops
+at ~1 MB.
+
+[arrow-rs#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+### B · Page-level dynamic prune at the row-group boundary
+
+<img src="/blog/images/sort-pushdown/future_page_level.png" alt="Page-level dynamic prune: extends #22450 to skip individual pages, not just whole row groups" width="100%" class="img-fluid"/>
+
+[#22450] prunes whole row groups at row-group boundaries. The
+finer-grained extension prunes whole **pages** within a surviving
+row group. The signal is the same dynamic filter, just re-applied
+at page granularity — for any page whose `max(col)` is already
+below the threshold, the filter column's bytes for that page can be
+skipped along with the projection columns.
+
+Today's page-level pruning runs once at file open using the static
+query predicate. Future B extends [#22450]'s "refresh at RG
+boundary" pattern to also rebuild the page-level filter with the
+live threshold, so upcoming row groups get tighter page selections
+mid-scan. Same arrow-rs API [#22450] already uses — no new
+primitive needed. Tracked in [apache/datafusion#23216].
+
+[apache/datafusion#23216]: https://github.com/apache/datafusion/issues/23216
+
+Conceptually this is the same idea as [#22450] stepped down one
+level: every level of the parquet hierarchy gets to chip off its
+share of the residue from the level above.
+
+## Acknowledgements
+
+Thank you to [@adriangb], [@alamb], [@xudong963], [@2010YOUY01], and
Review Comment:
It might also be worth giving a shout out to Massive for helping / sponsoring your work if appropriate.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
Review Comment:
Given how long the blog already is, I suspect many readers will not study potential future work. I recommend summarizing the future work in a sentence or two with links to the relevant arrow / datafusion tickets (and bonus points maybe to move these diagrams to those issues).
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
+
+Two complementary directions are open. The first needs an upstream
+arrow-rs primitive; the second is pure DataFusion plumbing on top
+of [#22450]:
+
+### A · Page-level `Exact` reverse · arrow-rs [#9937]
+
+<img src="/blog/images/sort-pushdown/reverse-scan.svg" alt="Row-group reverse (128 MB peak) vs page-level reverse (1 MB peak)" width="100%" class="img-fluid"/>
+
+[#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+Today's `DESC` query support lives in the `Inexact` path: the
+row-group reverse emits "RGs descending × rows ascending", which is
+close to DESC but not strictly so. `SortExec` stays.
+
+A page-level reverse primitive in arrow-rs would let the reader
+walk the parquet offset index in reverse — decoding each page
+forward, reversing its `RecordBatch`, and emitting before moving to
+the previous page. Peak buffer drops from ~128 MB (one
+row group) to ~1 MB (one page); per-page decode stays forward (RLE,
+dictionary, delta, and bit-packed encodings are all forward-only
+by construction — page *traversal* is what gets reversed). Once
+each batch already comes out in DESC order, `PushdownSort` can
+finally return `Exact` for `DESC`, the `SortExec` is removed
+outright, and `LIMIT N` becomes a static fetch.
+
+In flight upstream as [arrow-rs#9937]. The killer use case is
+**filtered reverse `TopK`** — e.g. `WHERE user_id = 42 ORDER BY ts
+DESC LIMIT 10`. You can't pre-compute a `RowSelection::with_limit`
+because matching rows are sparse; the only correct strategy is to
+stream pages backward, filter, and stop when 10 matches accumulate.
+Row-group reverse stops at ~128 MB granularity; page reverse stops
+at ~1 MB.
+
+[arrow-rs#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+### B · Page-level dynamic prune at the row-group boundary
+
+<img src="/blog/images/sort-pushdown/future_page_level.png" alt="Page-level dynamic prune: extends #22450 to skip individual pages, not just whole row groups" width="100%" class="img-fluid"/>
+
+[#22450] prunes whole row groups at row-group boundaries. The
+finer-grained extension prunes whole **pages** within a surviving
+row group. The signal is the same dynamic filter, just re-applied
+at page granularity — for any page whose `max(col)` is already
+below the threshold, the filter column's bytes for that page can be
+skipped along with the projection columns.
+
+Today's page-level pruning runs once at file open using the static
+query predicate. Future B extends [#22450]'s "refresh at RG
+boundary" pattern to also rebuild the page-level filter with the
+live threshold, so upcoming row groups get tighter page selections
+mid-scan. Same arrow-rs API [#22450] already uses — no new
+primitive needed. Tracked in [apache/datafusion#23216].
+
+[apache/datafusion#23216]: https://github.com/apache/datafusion/issues/23216
+
+Conceptually this is the same idea as [#22450] stepped down one
+level: every level of the parquet hierarchy gets to chip off its
+share of the residue from the level above.
+
+## Acknowledgements
+
+Thank you to [@adriangb], [@alamb], [@xudong963], [@2010YOUY01], and
+[@Dandandan] for reviewing the design and the patches across many
+iterations. The DataFusion community's willingness to engage deeply
+with optimizer changes — including the ones that touch foundational
+invariants like who-drives-the-decode-loop — is what made this work
+possible.
+
+[@alamb]: https://github.com/alamb
+[@adriangb]: https://github.com/adriangb
+[@xudong963]: https://github.com/xudong963
+[@2010YOUY01]: https://github.com/2010YOUY01
+[@Dandandan]: https://github.com/Dandandan
+
+## References
+
+Umbrella issue tracking the entire effort:
+
+* **[EPIC] Sort Pushdown · skip sorts and skip IO for ORDER BY / TopK queries: [apache/datafusion#23036](https://github.com/apache/datafusion/issues/23036)** — phase-by-phase status of all the PRs and follow-ups.
+
+Prior post this work builds on:
+
+* [Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries][dyn-filters-blog] — the dynamic filter primitive `TopK` uses.
+
+Landed PRs that make up the merged work:
+
+* `MinMaxStatistics` foundation: [apache/datafusion#9593](https://github.com/apache/datafusion/pull/9593)
+* `PushdownSort` rule + row-group reverse: [apache/datafusion#19064](https://github.com/apache/datafusion/pull/19064)
+* Reverse-output redesign: [apache/datafusion#19446](https://github.com/apache/datafusion/pull/19446), [apache/datafusion#19557](https://github.com/apache/datafusion/pull/19557)
+* Sort elimination via statistics: [apache/datafusion#21182](https://github.com/apache/datafusion/pull/21182)
+* `BufferExec` capacity for sort elimination: [apache/datafusion#21426](https://github.com/apache/datafusion/pull/21426)
+* Push-based parquet decoder (DataFusion owns the loop): [apache/datafusion#20839](https://github.com/apache/datafusion/pull/20839)
+* Morsel-style work scheduling: [apache/datafusion#21351](https://github.com/apache/datafusion/pull/21351)
+* Runtime reorder for `TopK` convergence: [apache/datafusion#21956](https://github.com/apache/datafusion/pull/21956)
+* **Runtime row-group dynamic pruning ([#22450])** — the centerpiece of this post.
+
+In flight / open:
+
+* Page-level reverse (arrow-rs): [apache/arrow-rs#9937](https://github.com/apache/arrow-rs/pull/9937), discussion in [apache/arrow-rs#9934](https://github.com/apache/arrow-rs/issues/9934)
+* `peek_next_row_group` API for per-RG `fully_matched` RowFilter skip (arrow-rs): [apache/arrow-rs#10158](https://github.com/apache/arrow-rs/pull/10158)
+* Page-level dynamic prune at RG boundary (Future B): [apache/datafusion#23216](https://github.com/apache/datafusion/issues/23216)
+* Per-RG `fully_matched` RowFilter skip on top of [#22450] (blocked on arrow-rs#10158): [apache/datafusion#23067](https://github.com/apache/datafusion/issues/23067)
+* Multi-column / function-wrapped stats reorder follow-ups: [apache/datafusion#22198](https://github.com/apache/datafusion/issues/22198)
+
+Concretely useful issues for new contributors:
Review Comment:
I recommend also adding a "get involved" section: https://datafusion.apache.org/blog/2026/05/28/writing-agent-skills/#get-involved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3545160273
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
Review Comment:
Follow-up on the reorder point — done in 9037e8a. Moved the Three-Layer Pruning section from after Runtime Row-Group Dynamic Pruning to right after How DataFusion Tracks Ordering, so the multi-layer framework is established before each layer is detailed. Promoted the topk_tpch benchmark to top-level ## since its parent section moved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3551805857
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,477 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
Review Comment:
What do you think about changing the title of the post to something like
```suggestion
title: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion
```
( I am absolutely trying to increase readership by making this sound more appealing than to the datafusion die hards)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5026662725
It's live: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
I'll post to the socials tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4898440339
Thanks @zhuqi-lucas -- I plan to re-read this blog shortly and provide another round of suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3545217618
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,543 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu and Andrew Lamb
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Qi Zhu](https://github.com/zhuqi-lucas), [Massive](https://www.massive.com/); [Andrew Lamb](https://github.com/alamb), [InfluxData](https://www.influxdata.com/)*
Review Comment:
Thanks @alamb I added your info, is it right? Let me know if it's not right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544701319
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — removed the `pruner_loop.png` figure; the surrounding markdown text already covers watch/rebuild/prune, so no overflow issue remains and the figure captions aren't needed.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Correction to my previous reply — I originally removed this figure, which was a bad call. Reverted in 126ad20; the figure is back. The overflow (Step 2 code text `into_builder().with_row_groups(...)` and Step 3 `Data → set_active_reader` bleeding outside their boxes) needs a diagram regeneration since only the PNG is checked in. The bottom-caption content (drain/decide/drive short-circuit behavior, Step 2 fires only at RG boundaries) is already covered in the surrounding blog text. Will regenerate the diagram in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544716601
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Correction to my previous reply — I originally removed this figure, which was a bad call. Reverted in 126ad20; the figure is back. The WATCH-circle overflow (`1 atomic load` cutoff) needs a diagram regeneration since only the PNG is checked in. The bottom captions (`big gear barely turns`, `errors fall back to keep`) are already covered by the surrounding blog text (or dropped in an earlier commit per your other feedback). Will regenerate the diagram in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5013926647
> > Otherwise all your edits LGTM — feel free to further tweak my wording if any of it reads off. Publish on Monday July 20 works for me; I'll ping Discord over the weekend for any last comments. Thanks again!
>
> Awesome -- I took another pass and wordsmithed (along with Claude)
>
> Looking forward to monday
Thanks @alamb for the final update!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5021550190
Shall we merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5029339324
> It's live: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
>
> I'll post to the socials tomorrow
Looks great, thanks @alamb !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544641569
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
Review Comment:
Done — dropped `now`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
Review Comment:
Done — `sorted on disk` → `sorted when stored`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
Review Comment:
Done — adopted suggested Iceberg bullet.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
Review Comment:
Done — linked Parquet `sorting_columns` to thrift; clarified `WITH ORDER` as DataFusion-specific.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
Review Comment:
Done — `full external SortExec` → `full sort`; `see` → `buffer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544654411
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
Review Comment:
Done — relabeled Layer 0/A/B → Layer 1/2/3 throughout.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
Review Comment:
Done — reframed as DF54 sort-opt disabled vs enabled.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
Review Comment:
Done — moved suite description into the intro.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
Review Comment:
Done — reframed positively: `3–4× on common time-series workloads with zero regressions`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
Review Comment:
Done — collapsed Future Directions to a summary sentence with issue links.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4828457400
### Community walkthrough · DataFusion China Meetup
I gave a walkthrough of this work at the **DataFusion China Meetup** ahead of opening this PR. Sharing the 24-slide deck here for reviewers / readers who want the talk-track version of the same material — same structure as the post (framework → Exact path → Inexact path → #22450 → future directions).
📎 **Slide deck** (`sort-pushdown-community-call.pdf`, 24 slides): _attached below — drag-and-drop preview_
Highlights from the talk that map to the post:
- The architecture-3-eras framing for who drives the I/O + decode loop (pre-#20839 / #20839 / #22450)
- The cascading-prune walkthrough — one row group fills `TopK`'s heap, every remaining RG cascades out at the next boundary
- The three-layer pruning stack (file + RG + row), all driven by the same `TopK` dynamic filter
- `topk_tpch` numbers (5 of 11 queries 3–4× faster, 0 regressions, total **−44%**)
Happy to take feedback on either the post or the deck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3540878839
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
Review Comment:
I really like the intro now
One nit is you can make maybe even stronger by removing the `now` so this post isn't quite as tied to some historic plan / process, but is describing a universal set of optimziations
```suggestion
**[Apache DataFusion] automatically takes advantage of sortedness in the
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
Review Comment:
This item seems a little different than the others that are explaining usescase when data is sorted on disk
maybe something more like
```suggestion
- Modern data lakes based on [Apache Iceberg] and similar formats
often store data in the order **it was written**, and resorting the
data is prohibitively expensive for many workloads.
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
Review Comment:
recommend a link to sorting columns defintion: https://github.com/apache/parquet-format/blob/8a5e04bdecf100e8e981daacfa117e8b5aadacb9/src/main/thrift/parquet.thrift#L1044
Also maybe make it clear that the with order thing is a datafusion thing
instead of
"and the table definition doesn't include a..."
Something like
"or the system wasn't informed about the ordering with a mechanism such as DataFusions `WITH_ORDER` clause"
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
Review Comment:
```suggestion
bias the order files are scanned so the *most-promising* data is read first —
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
Review Comment:
This refers to "TopK threshold" before that concept is introduced -- if we could describe this in more general terms here it might be better -- maybe something like "parquet decoder loop, re-check
dynamic predicate pruning if they have gotten more precise due to additional runtime information."
or something like this
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
Review Comment:
FWIW this can probably also be removed (or moved to a footnote) as an implementation detail and to shorten the article -- the section explains the optimization well without these caveats
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
Review Comment:
in my opinion, this will be confusing to many poeple as they won't know what a "per-partition" SortExec is -- it is a DataFusion specific thing
I suggest either
1. explain more how the SortExec works for multiple partitions or
2. (preferred) make this more generic and remove the details (remove straring at "Each per-partition...")
Instead of explaining the implementation details I suggest make the description higher level and refer readers to the PR for details.
Maybe something like
> Removing the `SortExec` looked like a pure win, but the first
multi-partition benchmarks showed something counter-intuitive: **some
queries got slower**. The root cause is that the removed `SortExec`
was doing two jobs — sorting *and* implicitly buffering.
The fix was to introduce explicit buffering in certain plans, see PR XYZ for details) as shown in
the following figure
...
(Figure)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
Review Comment:
Maybe use terminology less connected to past tech:
```suggestion
Many real datasets are at least partly sorted when stored:
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
Review Comment:
```suggestion
order* — deleting redundant sorts and biasing scan order toward the
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
Review Comment:
I don't have a great suggestion here, but this per file ordering vs listing ordering seems very specific to DataFUsion -- if we can find some way to make that clear it might be easier to understand
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
Review Comment:
I recommend making this more generic as it applies to all systems, not just DataFusion
```suggestion
In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
cost of a full sort — a pipeline-blocking operator that
must buffer every input row before emitting anything, dominating both
latency and peak memory on large scans.
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
Review Comment:
FWIW the references here to SortExec are good as this section is talking about DataFUsions internal implementation
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
Review Comment:
as above, i think using "sort" rather than `SortExec` would make it easier to see that this is not something only applicable to datafusion
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
Review Comment:
```suggestion
from the most-promising data, so the `TopK` threshold is more likely to tighten fast
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
Review Comment:
again here it is somewhat confusing to refer to TopK dynamic filtering without defining it -- maybe if you just included a link to the dynamic pruning blog post on the first reference it would be easier
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
Review Comment:
the fact that pushdown sort runs late seems like it is an irreelevant detail for this expanation -- we could probably remove it and shorten the cotnent.
Likewise the details that about how it walks the plan tree -- I don't think that adds anything more to the narrative and could be removed
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
Review Comment:
```suggestion
DataFusion could already recognize the *exact* sortedness case (declared
ordering + matching on-disk file list). It can now also recognize
sortedness when the **file list is in the wrong order** on disk, by using
the min/max row group statistics stored in the Parquet file (relevant PRs:
[apache/datafusion#19064][#19064] (rule scaffolding), and
[apache/datafusion#21182][#21182] (stats-based file reorder))
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
Review Comment:
I think it would help to add an explanatory description here about what this benchmark suite is all about -- something like
"To evaluate the effectiveness of eliminating sorts using statistics, we ran DataFusion's `sort_pushdown` benchmark suite"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4997811491
I took another pass and made it all the way to the end. I think it is looking pretty good now . Thank you @zhuqi-lucas for letting me go nuts
Some high level thoughts:
1. I think it would help the rigor of the presented numbers in the sort_tpch section if we provide/ links to the raw numbers underlying the chart
1. It might be worth considering the the appendixes (I moves the sections I felt were really focused on details to the end)
What i would like to suggest as next steps is:
1. You review my edits (there were a lot 😬 )
2. We can then post a note in discord / etc asking if anyone else wants to read / provide comments beforehand
3. Plan a publish date (maybe next Monday July 20th?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544701037
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — removed the `transition_anatomy.png` figure; the surrounding markdown text already covers the drain/decide/drive loop, so no overflow issue remains and the figure captions aren't needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544656209
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
+
+Two complementary directions are open. The first needs an upstream
+arrow-rs primitive; the second is pure DataFusion plumbing on top
+of [#22450]:
+
+### A · Page-level `Exact` reverse · arrow-rs [#9937]
+
+<img src="/blog/images/sort-pushdown/reverse-scan.svg" alt="Row-group reverse (128 MB peak) vs page-level reverse (1 MB peak)" width="100%" class="img-fluid"/>
+
+[#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+Today's `DESC` query support lives in the `Inexact` path: the
+row-group reverse emits "RGs descending × rows ascending", which is
+close to DESC but not strictly so. `SortExec` stays.
+
+A page-level reverse primitive in arrow-rs would let the reader
+walk the parquet offset index in reverse — decoding each page
+forward, reversing its `RecordBatch`, and emitting before moving to
+the previous page. Peak buffer drops from ~128 MB (one
+row group) to ~1 MB (one page); per-page decode stays forward (RLE,
+dictionary, delta, and bit-packed encodings are all forward-only
+by construction — page *traversal* is what gets reversed). Once
+each batch already comes out in DESC order, `PushdownSort` can
+finally return `Exact` for `DESC`, the `SortExec` is removed
+outright, and `LIMIT N` becomes a static fetch.
+
+In flight upstream as [arrow-rs#9937]. The killer use case is
+**filtered reverse `TopK`** — e.g. `WHERE user_id = 42 ORDER BY ts
+DESC LIMIT 10`. You can't pre-compute a `RowSelection::with_limit`
+because matching rows are sparse; the only correct strategy is to
+stream pages backward, filter, and stop when 10 matches accumulate.
+Row-group reverse stops at ~128 MB granularity; page reverse stops
+at ~1 MB.
+
+[arrow-rs#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+### B · Page-level dynamic prune at the row-group boundary
+
+<img src="/blog/images/sort-pushdown/future_page_level.png" alt="Page-level dynamic prune: extends #22450 to skip individual pages, not just whole row groups" width="100%" class="img-fluid"/>
+
+[#22450] prunes whole row groups at row-group boundaries. The
+finer-grained extension prunes whole **pages** within a surviving
+row group. The signal is the same dynamic filter, just re-applied
+at page granularity — for any page whose `max(col)` is already
+below the threshold, the filter column's bytes for that page can be
+skipped along with the projection columns.
+
+Today's page-level pruning runs once at file open using the static
+query predicate. Future B extends [#22450]'s "refresh at RG
+boundary" pattern to also rebuild the page-level filter with the
+live threshold, so upcoming row groups get tighter page selections
+mid-scan. Same arrow-rs API [#22450] already uses — no new
+primitive needed. Tracked in [apache/datafusion#23216].
+
+[apache/datafusion#23216]: https://github.com/apache/datafusion/issues/23216
+
+Conceptually this is the same idea as [#22450] stepped down one
+level: every level of the parquet hierarchy gets to chip off its
+share of the residue from the level above.
+
+## Acknowledgements
+
+Thank you to [@adriangb], [@alamb], [@xudong963], [@2010YOUY01], and
+[@Dandandan] for reviewing the design and the patches across many
+iterations. The DataFusion community's willingness to engage deeply
+with optimizer changes — including the ones that touch foundational
+invariants like who-drives-the-decode-loop — is what made this work
+possible.
+
+[@alamb]: https://github.com/alamb
+[@adriangb]: https://github.com/adriangb
+[@xudong963]: https://github.com/xudong963
+[@2010YOUY01]: https://github.com/2010YOUY01
+[@Dandandan]: https://github.com/Dandandan
+
+## References
+
+Umbrella issue tracking the entire effort:
+
+* **[EPIC] Sort Pushdown · skip sorts and skip IO for ORDER BY / TopK queries: [apache/datafusion#23036](https://github.com/apache/datafusion/issues/23036)** — phase-by-phase status of all the PRs and follow-ups.
+
+Prior post this work builds on:
+
+* [Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries][dyn-filters-blog] — the dynamic filter primitive `TopK` uses.
+
+Landed PRs that make up the merged work:
+
+* `MinMaxStatistics` foundation: [apache/datafusion#9593](https://github.com/apache/datafusion/pull/9593)
+* `PushdownSort` rule + row-group reverse: [apache/datafusion#19064](https://github.com/apache/datafusion/pull/19064)
+* Reverse-output redesign: [apache/datafusion#19446](https://github.com/apache/datafusion/pull/19446), [apache/datafusion#19557](https://github.com/apache/datafusion/pull/19557)
+* Sort elimination via statistics: [apache/datafusion#21182](https://github.com/apache/datafusion/pull/21182)
+* `BufferExec` capacity for sort elimination: [apache/datafusion#21426](https://github.com/apache/datafusion/pull/21426)
+* Push-based parquet decoder (DataFusion owns the loop): [apache/datafusion#20839](https://github.com/apache/datafusion/pull/20839)
+* Morsel-style work scheduling: [apache/datafusion#21351](https://github.com/apache/datafusion/pull/21351)
+* Runtime reorder for `TopK` convergence: [apache/datafusion#21956](https://github.com/apache/datafusion/pull/21956)
+* **Runtime row-group dynamic pruning ([#22450])** — the centerpiece of this post.
+
+In flight / open:
+
+* Page-level reverse (arrow-rs): [apache/arrow-rs#9937](https://github.com/apache/arrow-rs/pull/9937), discussion in [apache/arrow-rs#9934](https://github.com/apache/arrow-rs/issues/9934)
+* `peek_next_row_group` API for per-RG `fully_matched` RowFilter skip (arrow-rs): [apache/arrow-rs#10158](https://github.com/apache/arrow-rs/pull/10158)
+* Page-level dynamic prune at RG boundary (Future B): [apache/datafusion#23216](https://github.com/apache/datafusion/issues/23216)
+* Per-RG `fully_matched` RowFilter skip on top of [#22450] (blocked on arrow-rs#10158): [apache/datafusion#23067](https://github.com/apache/datafusion/issues/23067)
+* Multi-column / function-wrapped stats reorder follow-ups: [apache/datafusion#22198](https://github.com/apache/datafusion/issues/22198)
+
+Concretely useful issues for new contributors:
Review Comment:
Done — added `Get Involved` section.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
+groups, does that replace the `RowFilter` row-level prune that the
+`Inexact` path was already using?" **No** — the three layers stack,
+and they're driven by the **same** `TopK` dynamic filter. (The
+"Tier 1 / Tier 2" framing earlier maps to "Layer 0 / Layer A"
+below — same partition, different lens. Layer B is what runs on
+each row group after Layer A keeps it.)
+
+* **Layer 0 · file-level** (`file_pruner` + `EarlyStoppingStream`).
+ Cuts dead files before they're opened. The only layer that skips
+ parquet metadata I/O entirely. Already shipped before [#22450] —
+ this is Tier 1.
+* **Layer A · row-group-level** ([#22450]). Cuts dead row groups
+ inside open files at every row-group boundary. Bytes never
+ fetched, filter column never decoded. **This is the layer that
+ fills the Tier 2 gap** ("× no early stop yet" pre-[#22450]).
+* **Layer B · row-level** (`RowFilter`). For row groups that
+ survive Layer A, the filter is still evaluated row-by-row to
+ build a `RowSelection`. Rows that fail the predicate get their
+ *projection* columns short-circuited via arrow-rs's
+ `selects_any()`, but the *filter* column is necessarily read.
+ This layer has the highest residual cost (the filter column),
+ but also the finest granularity.
+
+The same dynamic filter drives all three. A single insertion into
+the `TopK` heap becomes a new threshold that Layer B applies
+per-row immediately (in the currently-open row group), and Layer A
+re-applies to remaining row groups at the next boundary. No layer
+subsumes another — Layer A prunes on metadata alone (never touching
+the filter column), while Layer B is finer-grained but has to read
+the filter column to decide.
+
+### Benchmark · `topk_tpch` (TPC-H SF1, `LIMIT 100`)
+
+<img src="/blog/images/sort-pushdown/topk_tpch_bench.png" alt="topk_tpch benchmark results: 5 of 11 queries 3-4× faster, 0 regressions, total -44%" width="100%" class="img-fluid"/>
+
+The [`topk_tpch`](https://github.com/apache/datafusion/blob/main/benchmarks/src/sort_tpch.rs) benchmark runs 11 TPC-H SF1 queries, all of the
+shape `ORDER BY ... LIMIT 100`, comparing `main` against the same
+branch with [#22450] enabled. Headline numbers:
+
+| Metric | Value |
+| ----------------------------------- | ---------------------------------- |
+| Total wall-clock (sum of 11 queries) | 248.8 ms → 139.1 ms (**−44%**) |
+| Queries with ≥2× speedup vs main | **5 of 11** (Q2, Q4, Q8, Q9, Q10) |
+| Queries with regression vs main | **0** |
+| Best single-query speedup | **~4×** |
+
+The five queries with significant speedups all use `l_orderkey`
+as the **leading** sort key — lineitem's physical sort key, a
+`BIGINT` with ~1.5M distinct values per SF1, so per-RG `min/max`
+ranges are cleanly disjoint and `Layer A` can cascade-prune
+aggressively. The non-winners (Q1, Q3, Q5, Q6, Q7, Q11) lead with
+`l_linenumber` (cardinality 7), `l_comment`, or `l_shipmode` —
+columns whose per-RG ranges overlap heavily because they're not the
+physical sort order. (Q5–Q7 still *include* `l_orderkey`, but only
+as a third-key tie-breaker — the leading key is what controls RG-level
+disjointness.) A tighter threshold doesn't translate into clean
+RG-level boundaries to prune at, so `Layer B` (row-level) still does
+its share of the work.
+
+The takeaway isn't "5 out of 11", it's "**zero regressions and
+no-op when the data doesn't help, 3–4× when it does**". The sweet
+spot — sort key aligned with the physical layout — is the common
+case for time-series, partitioned tables, and ingestion-ordered
+event logs.
+
+## Future Directions
+
+Two complementary directions are open. The first needs an upstream
+arrow-rs primitive; the second is pure DataFusion plumbing on top
+of [#22450]:
+
+### A · Page-level `Exact` reverse · arrow-rs [#9937]
+
+<img src="/blog/images/sort-pushdown/reverse-scan.svg" alt="Row-group reverse (128 MB peak) vs page-level reverse (1 MB peak)" width="100%" class="img-fluid"/>
+
+[#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+Today's `DESC` query support lives in the `Inexact` path: the
+row-group reverse emits "RGs descending × rows ascending", which is
+close to DESC but not strictly so. `SortExec` stays.
+
+A page-level reverse primitive in arrow-rs would let the reader
+walk the parquet offset index in reverse — decoding each page
+forward, reversing its `RecordBatch`, and emitting before moving to
+the previous page. Peak buffer drops from ~128 MB (one
+row group) to ~1 MB (one page); per-page decode stays forward (RLE,
+dictionary, delta, and bit-packed encodings are all forward-only
+by construction — page *traversal* is what gets reversed). Once
+each batch already comes out in DESC order, `PushdownSort` can
+finally return `Exact` for `DESC`, the `SortExec` is removed
+outright, and `LIMIT N` becomes a static fetch.
+
+In flight upstream as [arrow-rs#9937]. The killer use case is
+**filtered reverse `TopK`** — e.g. `WHERE user_id = 42 ORDER BY ts
+DESC LIMIT 10`. You can't pre-compute a `RowSelection::with_limit`
+because matching rows are sparse; the only correct strategy is to
+stream pages backward, filter, and stop when 10 matches accumulate.
+Row-group reverse stops at ~128 MB granularity; page reverse stops
+at ~1 MB.
+
+[arrow-rs#9937]: https://github.com/apache/arrow-rs/pull/9937
+
+### B · Page-level dynamic prune at the row-group boundary
+
+<img src="/blog/images/sort-pushdown/future_page_level.png" alt="Page-level dynamic prune: extends #22450 to skip individual pages, not just whole row groups" width="100%" class="img-fluid"/>
+
+[#22450] prunes whole row groups at row-group boundaries. The
+finer-grained extension prunes whole **pages** within a surviving
+row group. The signal is the same dynamic filter, just re-applied
+at page granularity — for any page whose `max(col)` is already
+below the threshold, the filter column's bytes for that page can be
+skipped along with the projection columns.
+
+Today's page-level pruning runs once at file open using the static
+query predicate. Future B extends [#22450]'s "refresh at RG
+boundary" pattern to also rebuild the page-level filter with the
+live threshold, so upcoming row groups get tighter page selections
+mid-scan. Same arrow-rs API [#22450] already uses — no new
+primitive needed. Tracked in [apache/datafusion#23216].
+
+[apache/datafusion#23216]: https://github.com/apache/datafusion/issues/23216
+
+Conceptually this is the same idea as [#22450] stepped down one
+level: every level of the parquet hierarchy gets to chip off its
+share of the residue from the level above.
+
+## Acknowledgements
+
+Thank you to [@adriangb], [@alamb], [@xudong963], [@2010YOUY01], and
Review Comment:
Done — added a Massive shout-out in Acknowledgements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4916388329
Thanks a lot @alamb — added you as second author, and I've addressed all comments. I'll do another self-review pass later. Let me know if there are any other issues. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4951625401
> I made it through the "The Exact Path: Sort Elimination via Statistics[¶](http://localhost:8000/blog/2026/07/05/sort-pushdown/#the-exact-path-sort-elimination-via-statistics)
>
> I will pick up The Inexact Path: Runtime Reorder for TopK and DESC[¶](http://localhost:8000/blog/2026/07/05/sort-pushdown/#the-inexact-path-runtime-reorder-for-topk-and-desc) next
Thanks @alamb for the great work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4880849259
Thanks @alamb for the very detailed review — hugely appreciated. I've addressed each of the 31 inline comments individually (see the reply threads).
**Summary of the pass:**
- **Concision**: `839 → 627 lines` (~25% shorter). Killed the TL;DR block, cut implementation-detail paragraphs, and trimmed the phase-landing narrative.
- **Reframed as a general DB technique**: opening is now **problem (partial sortedness / data lakes) → techniques (three complementary approaches) → results**, rather than the phase-by-phase "what we did" narrative.
- **Motivation**: added the Iceberg / data-as-written point up front, plus the stats-for-sort-vs-stats-for-filter contrast.
- **Trivial → exact**: renamed throughout, linked to @akurmustafa's [ordering-analysis blog](https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/) as background.
- **Links**: added inline links to `FileScanConfig`, `SortExec`, `SortPreservingMergeExec`, `BufferExec`, `FilePruner`, `PushdownSort` (docs.rs / GitHub source).
- **Pelican link-rendering issue**: converted all nine `` [`X`][ref] `` patterns to inline `` [`X`](url) `` so the code-in-link renders correctly.
- **Figures**: added `*Figure: ...*` captions under every diagram for skim-readability.
- **Exemplar query**: added the `ORDER BY ts DESC LIMIT 10` trades example when introducing Exact / Inexact runtime behaviour.
Ready for another pass whenever you have time. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4834047111
[sort-pushdown-community-call.pdf](https://github.com/user-attachments/files/29470687/sort-pushdown-community-call.pdf)
> @zhuqi-lucas is this blog ready for review?
Thanks @alamb , it should be ready, and mostly i copied content from my recent China Meetup slide above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3545698914
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,543 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu and Andrew Lamb
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Qi Zhu](https://github.com/zhuqi-lucas), [Massive](https://www.massive.com/); [Andrew Lamb](https://github.com/alamb), [InfluxData](https://www.influxdata.com/)*
Review Comment:
Yes, thank you
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,543 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu and Andrew Lamb
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Qi Zhu](https://github.com/zhuqi-lucas), [Massive](https://www.massive.com/); [Andrew Lamb](https://github.com/alamb), [InfluxData](https://www.influxdata.com/)*
Review Comment:
Yes, thank you
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,543 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu and Andrew Lamb
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Qi Zhu](https://github.com/zhuqi-lucas), [Massive](https://www.massive.com/); [Andrew Lamb](https://github.com/alamb), [InfluxData](https://www.influxdata.com/)*
Review Comment:
Yes, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544646589
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
Review Comment:
Done — high-level rewrite; details moved to PR #21426 link.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
Review Comment:
Done — added intro sentence explaining the `sort_pushdown` suite.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
Review Comment:
Done — condensed to a short paragraph, dropped the three-bullet enumeration.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
Review Comment:
Done — removed the `### When Inexact fires` heading.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
Review Comment:
Done — clarified the caption per suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544643150
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
Review Comment:
Done — `sorts, biasing` → `sorts and biasing`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
Review Comment:
Done — noted the per-file vs listing distinction is a DataFusion ListingTable concept.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
Review Comment:
Done — `SortExec` → `sort` in the three-technique bullets.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
Review Comment:
Done — applied suggested phrasing.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
Review Comment:
Done — rewrote in generic terms, linked to the dynamic-filters blog.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
Review Comment:
Done — rewrote in generic terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522647010
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
Review Comment:
Added a concrete `SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10` example with Exact / Inexact / no-info bullets.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
Review Comment:
Reworded — no longer claims 'same statistics'. The two paths are now described separately (plan-time non-overlap proof vs. runtime pruning against a live heap threshold).
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
Review Comment:
Merged the `PushdownSort` rule explanation into the tracking-ordering section as a single closing paragraph.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
Review Comment:
Now one sentence: `implemented across two PRs on PushdownSort: #19064 (scaffolding), #21182 (stats-based reorder)`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
Review Comment:
Done — `For example, consider three files a.parquet, b.parquet, c.parquet...`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
Review Comment:
Thanks — kept the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522646576
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
Review Comment:
Reframed as 'everything else — the messier real-world cases' with a bulleted list.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
Review Comment:
Restructured the opening as **problem → techniques → results**, and cut the linear phase-by-phase framing.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
Review Comment:
Linked from the abstract and multiple in-body mentions of `TopK`'s dynamic filter.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
Review Comment:
Trimmed to `2×–49× faster on ASC-LIMIT queries` — dropped the suite name.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
Review Comment:
Yes — that's [#21956](https://github.com/apache/datafusion/pull/21956) (`Runtime reorder for TopK convergence`), listed under Landed PRs. Called it out more explicitly in the Inexact section.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
Review Comment:
Removed.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
Review Comment:
Moved to Future Directions only. Intro no longer mentions it.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
Review Comment:
Removed the TL;DR block entirely; the 3-sentence abstract at the top now covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522647276
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
Review Comment:
Fixed — all backtick-in-link patterns rewritten as inline `[X](url)` (10 total after a follow-up sweep). Verified in the rendered HTML: 0 literal `[X][ref]` remaining.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4833998183
@zhuqi-lucas is this blog ready for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522646811
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
Review Comment:
That whole section was removed during the rewrite; the surviving mention uses `pipeline-blocking`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
Review Comment:
Removed the confusing bullet during the intro rewrite.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
Review Comment:
Added: 'Min/max statistics used for *predicate* pushdown are well-known... Using them to *reason about sort order* is less common.'
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
Review Comment:
Restructured so `WITH ORDER` only appears in the explicit 'what already worked' section, clearly separated from 'what was missing'.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
Review Comment:
Linked inline to `docs.rs/FileScanConfig`. Also switched other internals to inline docs.rs / GitHub links throughout.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
Review Comment:
Cut to a single sentence — implementation details dropped.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
Review Comment:
Linked to docs.rs for `SortPreservingMergeExec`, `SortExec`, `FileScanConfig`, `BufferExec`, `FilePruner`, and `PushdownSort`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522647255
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
Review Comment:
Cut the paragraph to essentials — three-step summary (sort file list / check adjacency / upgrade to Exact), with a link to `MinMaxStatistics` for readers who want more.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
Review Comment:
Fixed — all nine [\`X\`][ref] patterns rewritten as inline [\`X\`](url) so the Pelican markdown renderer doesn't drop them.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
+EnforceSorting + EnforceDistribution behaviour) can't remove it
+because the scan can't prove the ordering. The data is sorted on
+disk; the optimizer just can't tell.
+
+[`EnsureRequirements`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/ensure_requirements/mod.rs
+
+Stats-based sort elimination fixes this in `PushdownSort`, which
Review Comment:
Kept the phase2 non-overlap figure but stripped surrounding implementation detail; caption now just says `left case = safe to upgrade, right case = overlap, upgrade skipped, falls through to Inexact`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
+EnforceSorting + EnforceDistribution behaviour) can't remove it
+because the scan can't prove the ordering. The data is sorted on
+disk; the optimizer just can't tell.
+
+[`EnsureRequirements`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/ensure_requirements/mod.rs
+
+Stats-based sort elimination fixes this in `PushdownSort`, which
+runs late — after `EnsureRequirements` has already shaped the plan.
+When `PushdownSort` finds a `SortExec` above a file scan whose
+ordering was stripped (a `FileSource` `Unsupported` result), it
+does three things inside `FileScanConfig::try_pushdown_sort`:
+
+1. **Sort the file list by per-file statistics on the sort
+ column(s)** within each file group (the diagram above). The
+ pre-existing [`MinMaxStatistics`] helper reads each file's
+ `column_statistics[c].min_value` / `.max_value` for each sort
+ column `c`, then sorts the file list by the min row.
+ `sort_files_within_groups_by_statistics` does the per-group
+ orchestration and decides whether any group is non-overlapping
+ after the sort.
+2. **Check adjacency within each group**: walk each sorted file group
+ independently and ask whether `file[i].max ≤ file[i+1].min` for
+ every adjacent pair (touching at the boundary is fine — value `v`
+ showing up as the last row of one file and the first row of the
+ next still produces a sorted stream). The check is **per file
+ group**, not across groups; cross-group ordering is the job of
+ `SortPreservingMergeExec` at runtime (more on this below).
+3. **Upgrade `Unsupported` to `Exact`** when adjacency holds, the
+ table has a declared `output_ordering` (from `WITH ORDER` or
+ parquet `sorting_columns`), and the sort columns are null-free —
+ the last condition preserves `NULLS LAST`/`NULLS FIRST` semantics
+ across file boundaries. `PushdownSort` then removes the `SortExec`
+ itself and the plan becomes streamable.
+
+[`MinMaxStatistics`]: https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/statistics.rs
+
+One caveat that comes straight from `MinMaxStatistics`: the stats
+sort only fires when every `ORDER BY` expression is a plain column.
+`ORDER BY date_trunc('hour', ts)` silently skips the upgrade — there
+is no per-file min/max for the function output to compare against.
+Extending sort pushdown across monotonic function wrappers is one of
+the open follow-ups.
+
+(Runtime reorder covered later does handle some function-wrapped
+sorts via monotonicity inference — but stats-based sort elimination
+still needs a plain column.)
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid"/>
Review Comment:
Done — added `*Figure: ...*` captions under every diagram.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4917034932
> Thanks a lot @alamb — added you as second author, and I've addressed all comments. I'll do another self-review pass later. Let me know if there are any other issues. Thanks again!
Thank you @zhuqui-lucas -- I am taking a pass here too and will push some small edits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544716300
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Correction to my previous reply — I originally removed this figure, which was a bad call. Reverted in 126ad20; the figure is back. The overflow (Step 2 code text `into_builder().with_row_groups(...)` and Step 3 `Data → set_active_reader` bleeding outside their boxes) needs a diagram regeneration since only the PNG is checked in. The bottom-caption content (drain/decide/drive short-circuit behavior, Step 2 fires only at RG boundaries) is already covered in the surrounding blog text. Will regenerate the diagram in a follow-up.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Correction to my previous reply — I originally removed this figure, which was a bad call. Reverted in 126ad20; the figure is back. The WATCH-circle overflow (`1 atomic load` cutoff) needs a diagram regeneration since only the PNG is checked in. The bottom captions (`big gear barely turns`, `errors fall back to keep`) are already covered by the surrounding blog text (or dropped in an earlier commit per your other feedback). Will regenerate the diagram in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4915764999
Correction to the summary above — commit 9b48aa8 was a bad call (removed the two figures). Reverted in 126ad20, figures are back. The visual overflow and caption-in-figure feedback still needs addressing, but by regenerating the diagrams, not by dropping them. Will follow up separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544748521
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — regenerated `transition_anatomy.png` in 4e8a2d6. Widened the Step 2 / Step 3 code boxes so `into_builder().with_row_groups(...)` and `Data → set_active_reader` fit inside their boxes. Bottom captions removed (already covered in blog text).
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — regenerated `pruner_loop.png` in 4e8a2d6. Widened the WATCH shape so `1 atomic load` fits without cutoff. Dropped the `Errors fall back to keep` line (too much detail per your comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5021764774
> Shall we merge it?
Sure @alamb , let's merge it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5002602892
> Otherwise all your edits LGTM — feel free to further tweak my wording if any of it reads off. Publish on Monday July 20 works for me; I'll ping Discord over the weekend for any last comments. Thanks again!
Awesome -- I took another pass and wordsmithed (along with Claude)
Looking forward to monday
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #186:
URL: https://github.com/apache/datafusion-site/pull/186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3521386130
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
Review Comment:
I am not sure I would call the code for the existing sort orderings "trivial" 😆 -- I would probably use the term "exact" -- so like "been able to skip the sort in the *exact* case"
Maybe you could also link to this blog from @akurmustafa that explains what the existing function is : https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
Review Comment:
This is kind of a lot of detail -- I know what it means but many readers might not -- it might help to write this in non-datafusion technical terms. So instead of "when the catalog declares ..." something more like
"when the data on disk is perfectly sorted and table definition includes the ordering, DataFusion can avoid redundant sorts, as explained in the blog ...."
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
Review Comment:
Here are a few other motivating observations that might be worth working in
1. Data often has a partial sort order that is either not entirely sorted (e.g. ingestion order vs data order) or else the database doesn't know about the existing ordering
2. Modern data lakes based on Apache Icerberg and others often have to work with data as it is / was written and don't have the luxury of making a second copy, or resorting -- so being able to automatically exploit existing sortedness is important
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
Review Comment:
And then in this next sentence, rather than calling it "hard" maybe call it "another" case and describe in terms of user visible behavior: "This blog is about taking advantage of sorting even when the files may not be perfectly sorted and DataFusion is not told about any ordering up front"
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
Review Comment:
I suggest you start this blog with a 2-3 sentence abstract / summary of its contents, so that a potential reader can quickly decide if they want to read it or not
Maybe something like
"DataFusion now automatically takes advantage of sortedess in the data, even when the data is only partially sorted and DataFusion does not know the sort order ahead of time. This blog explains why this is an important usecase and how Datafusion achieves this, through a combination of pushing down sort information, and adaptive runtime reordering and pruning"
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
Review Comment:
Did we also merge the "reorder row groups to try and better take advantage of these optimizations" PR too?
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
Review Comment:
I think a more databasy term is "It is a pipeline blocker"
```rust
It is pipeline blocking operation — no row can leave until every input
```
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
Review Comment:
This note might be better left in a "future work" section at the end rather than in the intro
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
Review Comment:
This isa lot of deaail - I think this could be made a lot more concise by remvoing the implementation details. SOmething like
**`Exact`** — the optimizer is *certain* the output is in this order and removes redundant `SortExec`s entirely. `
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
Review Comment:
This paragraph seems like an implementation detail and could be removed without changing the overall narrative
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
Review Comment:
this seems somewhat redundant with the previous section -- I think if you combined them it would be clearer and more concise
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
Review Comment:
It isn't clear to me from the text how Inexact uses the **same** min/max statistics... The Top-K / dynamc filter uses the same idea (track the maximum min) but i don't think it is the same statistics
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
Review Comment:
"For example: "
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
Review Comment:
I think this is a lot of implementation detail that will not mean much to anyone else -- just saying "previously DataFusion could not recognize the sort order, but now it can" is probably enough, with a link to the code for anyone who wants more details
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
+EnforceSorting + EnforceDistribution behaviour) can't remove it
+because the scan can't prove the ordering. The data is sorted on
+disk; the optimizer just can't tell.
+
+[`EnsureRequirements`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/ensure_requirements/mod.rs
+
+Stats-based sort elimination fixes this in `PushdownSort`, which
+runs late — after `EnsureRequirements` has already shaped the plan.
+When `PushdownSort` finds a `SortExec` above a file scan whose
+ordering was stripped (a `FileSource` `Unsupported` result), it
+does three things inside `FileScanConfig::try_pushdown_sort`:
+
+1. **Sort the file list by per-file statistics on the sort
+ column(s)** within each file group (the diagram above). The
+ pre-existing [`MinMaxStatistics`] helper reads each file's
+ `column_statistics[c].min_value` / `.max_value` for each sort
+ column `c`, then sorts the file list by the min row.
+ `sort_files_within_groups_by_statistics` does the per-group
+ orchestration and decides whether any group is non-overlapping
+ after the sort.
+2. **Check adjacency within each group**: walk each sorted file group
+ independently and ask whether `file[i].max ≤ file[i+1].min` for
+ every adjacent pair (touching at the boundary is fine — value `v`
+ showing up as the last row of one file and the first row of the
+ next still produces a sorted stream). The check is **per file
+ group**, not across groups; cross-group ordering is the job of
+ `SortPreservingMergeExec` at runtime (more on this below).
+3. **Upgrade `Unsupported` to `Exact`** when adjacency holds, the
+ table has a declared `output_ordering` (from `WITH ORDER` or
+ parquet `sorting_columns`), and the sort columns are null-free —
+ the last condition preserves `NULLS LAST`/`NULLS FIRST` semantics
+ across file boundaries. `PushdownSort` then removes the `SortExec`
+ itself and the plan becomes streamable.
+
+[`MinMaxStatistics`]: https://github.com/apache/datafusion/blob/main/datafusion/datasource/src/statistics.rs
+
+One caveat that comes straight from `MinMaxStatistics`: the stats
+sort only fires when every `ORDER BY` expression is a plain column.
+`ORDER BY date_trunc('hour', ts)` silently skips the upgrade — there
+is no per-file min/max for the function output to compare against.
+Extending sort pushdown across monotonic function wrappers is one of
+the open follow-ups.
+
+(Runtime reorder covered later does handle some function-wrapped
+sorts via monotonicity inference — but stats-based sort elimination
+still needs a plain column.)
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid"/>
Review Comment:
The image is great -- I recommend adding a figure label to explain it to someone who is just skimming (not reading hte article linerly)
<img width="1000" height="403" alt="Image" src="https://github.com/user-attachments/assets/779c050a-4855-4b5f-a40a-aaec2c53e79f" />
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
Review Comment:
I think this paragrpah is redundant
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
Review Comment:
See overall comment about linear story telling vs problem/resolution structure
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
Review Comment:
I think you can make this more concise -- e.g. by not naming the benchmark suite (as you haven't yet introduced them yet anyways). Just saying "2.x-49x faster for certain benchmarks" is probablt enough
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
Review Comment:
I don't think this is really a TLDR -- it is quite long and it also repeats information from the previous intro
<img width="1124" height="685" alt="Image" src="https://github.com/user-attachments/assets/3affadcd-d81d-4676-95a1-2220e34b64b3" />
I suggest removing this section and adding a 2-3 sentence summary at the very top
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
Review Comment:
I don't understand this point - those operators need sorted data so sorting isn't a wasted cost. If the data is already sorted then this would be covered by the first point
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
Review Comment:
I suggest you link to the relevant blog post about topk / dynamic filters: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
Review Comment:
It might also be useful to point out that it is common to use the statistics for filter based pruning, but
that using them to infer sortedness / partial sortedness for sort based optimizations is not as common
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
Review Comment:
I thought this whole post was about taking advantge of sort order that Datafusion **didn't** already know about, and if so this mention of `WITH ORDER` is confusing
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
Review Comment:
Suggest linking to the source / deifning wht these operators are
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
Review Comment:
If you are going to refer to DataFusion's internals, I recommend linking to the docs.rs or github url link.
Also, as I mentioned above, I think this blog will have a larger audience if you minimize the DataFusion specific terminology as much as possible -- for example here you could say
> DataFusion tracks any pre-exisitng ordering using the `FileScanConfig::output_ordering` flag.
> The pre existing ordering can be either:
and make sure to link to a definition of the `FileScanConfig`
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
Review Comment:
I found the mention of "the `LIMIT`" here a little confusing as it wasn't previously referred to.
Perhaps you could define an exemplar query like
> For example, let's consider a query such as the following to retrieve the 10 most recent trades:
>
> ```sql
> SELECT timestamp, trade, amount ORDER BY timestamp DESC LIMIT 10
> ```
>
> Without any knowledge of the data sort order, DataFusion will scan all he data and
> use a Top-K operator. However, given sort knowledge:
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
Review Comment:
I think the details of how the work landed could be omitted for brevity -- like maybe just say " What was missing was
sort elimination when the file list was in the **wrong order**, which was implemented in #19064 and #21182
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
Review Comment:
BTW this doesnt' seem to be a link
<img width="1000" height="403" alt="Image" src="https://github.com/user-attachments/assets/4afbee8f-98be-4fc1-9859-38430cddd339" />
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
Review Comment:
this is a good example
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
Review Comment:
I also recommend linking to the docs about `WITH ORDER` as it isn't SQL standard (it is a datafusion specific thing)
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
+**removes the redundant `SortExec`**. The hard cases were
+everything just slightly looser — files in the wrong on-disk order,
+declared but overlapping ranges, no declaration at all,
+`ORDER BY ... DESC` on ASC-sorted data — which all paid a full
+external sort across the entire scan. CPU wasted. Memory wasted.
+Streaming defeated.
+
+[Apache DataFusion]: https://datafusion.apache.org/
+
+This post walks through the **sort pushdown** series of work — a
+multi-release effort spanning DataFusion **v52 through v55+** — that
+closed that gap on three layers at once:
+
+1. **`Exact` path** — delete the `SortExec` entirely when statistics
+ prove the scan is already ordered. Headline: **2.1×–49×** on the
+ `sort_pushdown` benchmark suite.
+2. **`Inexact` path** — keep the `SortExec`, but read the
+ most-promising data first so `TopK`'s dynamic filter tightens fast
+ and stale row groups get pruned by statistics.
+3. **Runtime row-group pruning** ([#22450]) — the latest piece, which
+ re-checks the dynamic filter at every row-group boundary and
+ physically removes pruned row groups from the open `RecordBatch`
+ stream. Headline: **5 of 11** `topk_tpch` queries run **3–4× faster**
+ with zero regressions; total benchmark time drops by **−44%**.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+
+Together these compose into a **three-layer pruning stack** —
+file-level, row-group-level, row-level — all driven by the same
+`TopK` dynamic filter. **None of the runtime parts of this post
+would be possible without TopK's dynamic filter pushdown** — the
+primitive that lets `TopK`'s threshold reach the parquet scanner
+mid-execution. The earlier [Dynamic Filters][dyn-filters-blog]
+post covers that primitive in detail and is recommended background
+for what follows.
+
+The page-level reverse primitive we are adding upstream in
+[arrow-rs] will push the `DESC` gains further still.
+
+[arrow-rs]: https://github.com/apache/arrow-rs
+
+## TL;DR
+
+* DataFusion can now **eliminate `SortExec` in far more cases** —
+ even when files are listed in the wrong order on disk — **read the
+ most-promising data first** when sorts can't be eliminated, and
+ **drop row groups mid-scan** as `TopK`'s heap converges.
+* What landed across v52–v55+:
+ * **The `PushdownSort` rule** — a physical optimizer rule that
+ asks each `ExecutionPlan` "can you produce output in *this*
+ ordering?" and uses the `Exact` / `Inexact` / `Unsupported`
+ answer to decide whether to delete the surrounding `SortExec`,
+ leave it in place with a hint, or give up.
+ * **Sort elimination via statistics (`Exact` path)** —
+ `PushdownSort` sorts files within each partition by Parquet
+ `min/max` statistics and, when the resulting ranges are provably
+ non-overlapping, upgrades the source's ordering claim from
+ `Unsupported` to `Exact` and **removes the `SortExec`** that
+ earlier passes left in the plan. `BufferExec` keeps multi-partition
+ plans from regressing when the eliminated `SortExec` was also
+ serving as an implicit buffer.
+ * **Runtime reorder (`Inexact` path)** — when the leading sort key
+ is a plain column (or the reversed source ordering satisfies the
+ request), the scan reorders files and row groups by `min/max`
+ stats so the most-promising data is read first; for `DESC`
+ requests it additionally flips iteration. The verdict is
+ `Inexact`, so `SortExec` stays in place, but `TopK`'s dynamic
+ filter tightens fast and the rest is pruned. File-level early
+ stop has been working for a while; **row-group-level early stop
+ was the missing piece until [#22450]**.
+ * **Runtime row-group dynamic pruning ([#22450])** — inside the
+ decoder loop, at every row-group boundary, the live `TopK`
+ threshold is converted into a fresh `PruningPredicate` and used
+ to drop subsequent row groups before any bytes are fetched.
+ A single row group can fill `TopK`'s heap, snap the threshold,
+ and cascade-prune every remaining row group in one pass — no
+ I/O, no decompress, no decode, not even the filter column.
+* Benchmarks:
+ * **`Exact` path on `sort_pushdown`**: `ORDER BY ... LIMIT` runs
+ **27× and 49× faster**; full `ORDER BY` scans run **~2×** faster.
+ * **[#22450] on `topk_tpch`** (TPC-H SF1, all `ORDER BY ... LIMIT 100`):
+ 5 of 11 queries get **3–4× faster**, **0 regressions**, total
+ runtime drops **−44%** (248.8 ms → 139.1 ms).
+
+## Why Sort Pushdown Matters
+
+`SortExec` is one of the most expensive operators in a query plan.
+It is blocking by construction — no row can leave until every input
+row has been seen and compared — so it tends to dominate both latency
+and peak memory. The cost gets paid even when:
+
+* the file is already ordered by the sort key (very common for
+ timestamp columns);
+* the query only needs the top *N* rows (`ORDER BY ts LIMIT 100`), in
+ which case full sort + truncate is wildly wasteful;
+* the next operator (`SortPreservingMergeExec`, `SortMergeJoinExec`,
+ a window function) was going to consume ordered input anyway.
+
+The data DataFusion needs to avoid this work is **already in the file
+metadata**. Parquet writers can record per-column statistics (`min`,
+`max`) at the row-group level. Files written by Spark, DuckDB,
+arrow-rs, and others routinely include them. And explicit `WITH ORDER`
+clauses in DataFusion's SQL `CREATE EXTERNAL TABLE` give the optimizer
+a direct ordering hint. The job of sort pushdown is to **use that
+information** — at plan time when possible, and at runtime when only
+the live state of the query can tell us what's still worth reading.
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+Each `FileScanConfig` carries an `output_ordering` — the ordering
+that the optimizer is willing to claim for the scan's output. There
+are two flavours:
+
+* **`Exact`** — the optimizer is *certain* the output is in this order.
+ Optimizer rules treat an `Exact` ordering as a proof and **remove
+ the surrounding `SortExec`** entirely. [`EnsureRequirements`] does
+ this when the scan already declared `Exact` from the start; the
+ [`PushdownSort`] rule covered in this post does the same after
+ upgrading from `Unsupported` via stats-based reorder.
+* **`Inexact`** — the optimizer *believes* the output is probably
+ ordered, but cannot prove it. Downstream operators like
+ `SortPreservingMergeExec` can still benefit from this hint, but the
+ explicit `SortExec` stays for safety.
+
+[`PushdownSort`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs
+
+A helper called `validated_output_ordering()` is the gatekeeper. It
+walks the list of files inside a partition, checks whether the
+declared per-file ordering is consistent with the file order on disk,
+and either confirms the ordering or **strips it entirely** if it
+sees something ambiguous (e.g. file `b` comes before file `a` in the
+file list but file `a`'s range comes first).
+
+### `Exact` and `Inexact` at runtime
+
+`Exact` and `Inexact` lead to different runtime behaviour, and
+distinguishing them up front makes the rest of this post easier to
+follow:
+
+* With **`Exact`**, the `SortExec` is removed and the LIMIT becomes
+ a **static fetch** on the source. The reader stops the moment the
+ requested number of rows has been emitted — early termination
+ at batch granularity, no dynamic state needed.
+* With **`Inexact`**, the `SortExec` stays in place. The LIMIT
+ materialises inside the sort as a `TopK` heap of size K. `TopK`
+ exposes a [**dynamic filter**][dyn-filters-blog] — a runtime
+ expression of the form *"only rows that could still beat the
+ current K-th-best value are worth considering"* — and pushes it
+ back to the parquet scanner. As more data is processed and the
+ heap tightens, the filter's threshold tightens with it, and entire
+ row groups can be skipped by checking the live threshold against
+ the row group's min/max statistics. (See the earlier
+ [dynamic filters][dyn-filters-blog] post for the full background
+ on this mechanism.)
+
+Both paths use the same underlying min/max statistics, but for
+different purposes: `Exact` uses them at plan time to prove
+non-overlap and justify removing the sort; `Inexact` uses them at
+runtime to skip data that can no longer improve the heap.
+
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## The `PushdownSort` Rule
+
+The **`PushdownSort`** physical optimizer rule asks each
+`ExecutionPlan` two questions:
+
+1. "Can you produce output in *this* ordering?"
+2. "If yes, please rearrange yourself so that it actually does."
+
+The answer is one of `Exact`, `Inexact`, `Unsupported`. `Exact`
+means the surrounding `SortExec` can be deleted entirely; `Inexact`
+means the source will read the data in a near-sorted order so
+`TopK` and other consumers benefit, but `SortExec` stays for
+strict correctness; `Unsupported` means the source can't help and
+`SortExec` does the full job. The rest of this post is what each
+merged capability does on top of this protocol — first the `Exact`
+path, then the `Inexact` path, then the runtime row-group prune
+that `Inexact` was missing.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="Sort elimination: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid"/>
+
+A note on what's actually new here: the *trivial* sort-elimination
+case (declared ordering + matching on-disk file list) has always
+worked via the `EnsureRequirements` rule. What was missing was
+sort elimination when the file list was in the **wrong order** on
+disk. `PushdownSort` landed across two phases — [Phase 1
+(#19064)][#19064] introduced the rule scaffolding and reverse
+iteration (the latter is used by the `Inexact` path covered
+later); [Phase 2 (#21182)][#21182] added stats-based file reorder
+so the rule can also produce `Exact` for the wrong-disk-order case.
+The motivating scenario for Phase 2:
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+* Three files: `a.parquet`, `b.parquet`, `c.parquet`.
+* Each declares `WITH ORDER (ts ASC)`.
+* Internally each file *is* sorted by `ts`.
+* But they were written by different ingestion jobs and end up listed
+ in the **wrong order** on disk (e.g. alphabetical by name, not by
+ time).
+
+`validated_output_ordering()` looks at this, sees that the
+file-internal ordering disagrees with the file-list order, and
+**strips the ordering entirely**. From the optimizer's point of view
+the scan now has no declared ordering. Physical planning translates
+the user's `ORDER BY` into a `SortExec`, and the earlier pipeline
+rule [`EnsureRequirements`] (which consolidates the old
+EnforceSorting + EnforceDistribution behaviour) can't remove it
+because the scan can't prove the ordering. The data is sorted on
+disk; the optimizer just can't tell.
+
+[`EnsureRequirements`]: https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/ensure_requirements/mod.rs
+
+Stats-based sort elimination fixes this in `PushdownSort`, which
Review Comment:
This is also full of implementation details -- I suggest you focus on just explaining the conditions under which the transformation can be done (non overlapping min/max boundaries)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3557200000
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,477 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
Review Comment:
Good suggestion! Committed this suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4998603495
Thanks so much @alamb — went through everything and your edits read really well. The restructure (moving buffering / decoder-loop into appendices, tightening the intro, expanding Related Work) makes the whole post much clearer, and the phrasing is consistently better than what I had.
Pushed one tiny follow-up commit ([1666952](https://github.com/apache/datafusion-site/pull/186/commits/1666952)) with three small technical fixes on top of your work:
1. **Row-Level Early Stopping paragraph**: attributed the "skip projection column fetch" behavior to arrow-rs's `RowFilter` (where it actually lives) and clarified that #22450's contribution is keeping the threshold up-to-date at row-group boundaries so this fires more often.
2. **"In DataFusion 54"** → "Over the last few DataFusion releases" — #22450 merged 2026-06-22, after `54.0.0` was tagged, so it will ship in a later release; safer to not commit to a version.
3. **Decision-tree figure alt-text**: `try_pushdown_sort` → `PushdownSort` to match the struct name in the surrounding prose.
On your two suggestions:
- **Raw numbers for sort_tpch**: agreed, I'll add per-query timing table like the sort_pushdown section has. Will do in a follow-up.
- **Appendix structure**: happy with your move, keeps the main narrative flowing.
+1 on aiming for **Monday July 20** publish; I'll ping on Discord over the weekend once the per-query table lands so anyone else who wants a last read can weigh in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4998606114
Thanks so much for the deep pass @alamb — the restructure reads great, and the Related Work section is stronger than anything I would've written on my own. Really appreciate the time.
I made three tiny tweaks in [1666952](https://github.com/apache/datafusion-site/commit/1666952):
- Row-Level Early Stopping paragraph: moved the "skip projection column fetch" attribution to arrow-rs's `RowFilter` (where it actually lives) and clarified that #22450 makes it fire more often by keeping the threshold current.
- Dropped "In DataFusion 54" from the conclusion — #22450 merged after 54.0.0 was tagged, so it'll ship in a later release; used "Over the last few DataFusion releases" instead.
- Decision-tree figure alt-text: `try_pushdown_sort` → `PushdownSort` for consistency with the prose.
Otherwise all your edits LGTM. Happy to go with **Monday July 20** as the publish date — I'll ping Discord over the weekend for any last comments before then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-5024993790
Ah, I see somehow my approval got cleared. Sorry about that I didn't realize we were waiting on me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522646380
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
Review Comment:
Done — added a 3-sentence abstract at the top.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
Review Comment:
Added: time-series files, event logs, partition-keyed tables, and Iceberg / data-lake data-as-written are now called out.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
Review Comment:
Renamed to *exact* and linked to @akurmustafa's ordering-analysis post.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
Review Comment:
Added a link to the WITH ORDER docs.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
+or parquet `sorting_columns`) **and** the on-disk file listing
+already matches that order, the `EnsureRequirements` rule sees that
+the scan's `output_ordering` already satisfies the request and
Review Comment:
Rewrote using plainer terms — no more `catalog declares`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3522646487
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,839 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+Many [Apache Parquet] datasets are already sorted on disk. Time-series
+files are usually written in ingestion-time order. Event logs are sharded
+and sorted by event id. Partitioned tables come with a natural ordering
+implied by the partition key. The information about that ordering is
+sitting right there in the file metadata.
+
+[Apache Parquet]: https://parquet.apache.org/
+
+[Apache DataFusion] has always been able to skip the sort in the
+*trivial* case: when the catalog declares an ordering (`WITH ORDER`
Review Comment:
Renamed to *exact* and linked to akurmustafa's ordering-analysis post.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#issuecomment-4920996125
> Thanks again @zhuqi-lucas -- I pushed a few more commits to try and make this more concise and focused on the main narrative
>
> However, I now need to go focus on other things for a while
>
> I think it would be worth reviewing the figures again (and pulling out some of the in-diagram labels to text in the post, and making sure they don't overflow, etc). I'll try and find tommorrow
Thanks @alamb !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544649524
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
Review Comment:
Done — named `file_d` / `file_c` and defined `low-value` (smaller sort key for ASC).
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
Review Comment:
Done — renamed to `Row Group Filter early stop`; moved PR #22450 into body.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
Review Comment:
Done — rewrote alt text; dropped `Tier 2` / `pre-#22450`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
Review Comment:
Done — replaced with a high-level paragraph linking to late-materialization blog.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
Review Comment:
Done — renamed to `Runtime Row-Group Dynamic Pruning`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544651275
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
Review Comment:
Done — merged that paragraph into the next section intro.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
Review Comment:
Done — reframed as `DataFusion now also re-checks…`.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
Review Comment:
Done — added the `only when the filter has updated since the initial check` qualifier.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
Review Comment:
Done — removed the three-eras evolution history; kept the current loop only.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
Review Comment:
Noted — figure annotation, will update the diagram separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #186:
URL: https://github.com/apache/datafusion-site/pull/186#discussion_r3544653057
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
Review Comment:
Noted — figure annotation, will update the diagram separately.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
Review Comment:
Done — applied suggested phrasing.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
Review Comment:
Done — renamed to `Example of pruning row groups using TopK dynamic predicates` + added intro.
##########
content/blog/2026-07-05-sort-pushdown.md:
##########
@@ -0,0 +1,625 @@
+---
+layout: post
+title: Sort Pushdown in DataFusion: Skip Sorts, Skip Decode, Skip I/O
+date: 2026-07-05
+author: Qi Zhu
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*Qi Zhu, [Massive](https://www.massive.com/)*
+
+**[Apache DataFusion] now automatically takes advantage of sortedness in the
+data — even when the data is only *partially* sorted, and even when
+DataFusion has not been told about the ordering ahead of time.** This post
+explains why that matters and walks through how DataFusion achieves it,
+through a combination of plan-time sort pushdown, runtime scan reordering,
+and mid-scan row-group pruning driven by [dynamic filters][dyn-filters-blog].
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[dyn-filters-blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/
+
+## Why sort pushdown matters
+
+Many real datasets are at least partly sorted on disk:
+
+- Time-series files are written in ingestion-time order.
+- Event logs are sharded and sorted by event id.
+- Partitioned tables have a natural ordering by partition key.
+- Modern data lakes based on [Apache Iceberg] and similar formats
+ often have to work with data **as it was written** — resorting the
+ whole table isn't an option.
+
+But that "pre-existing sortedness" is only useful if the query engine can
+**notice** it and **use** it. Two common failure modes:
+
+1. The engine doesn't know about the ordering — the writer didn't set
+ Parquet `sorting_columns`, and the table definition doesn't include a
+ [`WITH ORDER`](https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table) clause.
+2. The engine knows the *per-file* ordering, but the file *listing* on
+ disk is in a different order, so global sortedness can't be proven at
+ plan time.
+
+In both cases, an `ORDER BY` or `ORDER BY ... LIMIT N` query pays the
+cost of a full external `SortExec` — a pipeline-blocking operator that
+must see every input row before emitting anything, dominating both
+latency and peak memory on large scans.
+
+Min/max statistics used for *predicate* pushdown are well-known and
+widely implemented across databases. Using them to *reason about sort
+order* — deleting redundant sorts, biasing scan order toward the
+most-promising data — is less common. This post is about how DataFusion
+does the latter.
+
+[Apache Iceberg]: https://iceberg.apache.org/
+
+## What DataFusion could already do — and what was missing
+
+DataFusion has always been able to skip the sort in the **exact** case,
+using the machinery covered in [@akurmustafa's earlier post on
+ordering analysis][ordering-analysis]: when the table definition
+declares an ordering (via `WITH ORDER` or Parquet `sorting_columns`)
+**and** the on-disk file listing already matches that order, the
+existing `EnsureRequirements` rule sees that the scan's
+`output_ordering` satisfies the request and **removes the redundant
+`SortExec`** entirely.
+
+This post is about **everything else** — the messier real-world cases
+where sortedness exists but isn't provable up front:
+
+- Files listed in the "wrong" order on disk (each file internally
+ sorted, but the listing doesn't match).
+- Declared ordering with **overlapping** ranges across files.
+- **No** declared ordering at all.
+- `ORDER BY ... DESC` on ASC-sorted data.
+
+Three complementary techniques close each gap:
+
+1. **Statistics-based sort elimination** (`Exact` path). Extend the
+ optimizer to prove ordering from min/max statistics after
+ reordering the file list, then delete the `SortExec` entirely.
+2. **Runtime scan reorder** (`Inexact` path). Keep the `SortExec`, but
+ bias scan order so the *most-promising* data is read first —
+ `TopK`'s [dynamic filter][dyn-filters-blog] tightens quickly and
+ downstream data is pruned by statistics before it's read.
+3. **Runtime row-group dynamic pruning** ([#22450]). Inside the
+ parquet decoder loop, re-check the live `TopK` threshold at every
+ row-group boundary and physically remove pruned row groups before
+ any bytes are fetched.
+
+Together these compose into a **three-layer pruning stack**
+(file-level, row-group-level, row-level), all driven by the same
+`TopK` dynamic filter. Headline results:
+
+- **Sort elimination**: 2×–49× faster on ASC-LIMIT queries where the
+ file list was in the wrong disk order.
+- **Runtime row-group pruning ([#22450])**: 5 of 11 `topk_tpch`
+ queries run 3–4× faster with zero regressions; total runtime drops
+ −44%.
+
+The rest of this post walks through each technique in turn.
+
+[#22450]: https://github.com/apache/datafusion/pull/22450
+[#20839]: https://github.com/apache/datafusion/pull/20839
+[Apache Parquet]: https://parquet.apache.org/
+[ordering-analysis]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/
+
+## How DataFusion Tracks Ordering
+
+<img src="/blog/images/sort-pushdown/plan-diff.svg" alt="EXPLAIN before / after: SortExec eliminated once ordering is Exact" width="100%" class="img-fluid"/>
+
+DataFusion's [`FileScanConfig`](https://docs.rs/datafusion-datasource/latest/datafusion_datasource/file_scan_config/struct.FileScanConfig.html) carries an ordering claim for
+each scan's output, which is one of:
+
+- **`Exact`** — the optimizer is *certain* the output is in this order,
+ and removes redundant [`SortExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort/struct.SortExec.html) operators entirely.
+ `LIMIT N` becomes a static fetch on the source (the reader stops the
+ moment N rows are emitted).
+- **`Inexact`** — the optimizer believes the output is probably ordered
+ but cannot prove it. Downstream operators like
+ [`SortPreservingMergeExec`](https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/sorts/sort_preserving_merge/struct.SortPreservingMergeExec.html) can still benefit, but the
+ explicit `SortExec` stays for correctness. In this case `TopK`'s
+ [dynamic filter][dyn-filters-blog] tightens as the heap fills, and
+ data whose min/max cannot beat the threshold is pruned before it is
+ fully read.
+
+For example, given a query that returns the 10 most recent trades:
+
+```sql
+SELECT ts, symbol, amount FROM trades ORDER BY ts DESC LIMIT 10;
+```
+
+- With no ordering knowledge, DataFusion scans everything and uses a
+ `TopK` heap to keep the running best 10.
+- With **`Exact`** ordering, DataFusion drops the sort entirely and
+ stops reading after emitting 10 rows.
+- With **`Inexact`** ordering, the `SortExec` stays but scans start
+ from the most-promising data, so the `TopK` threshold tightens fast
+ and the rest is pruned by statistics.
+
+The optimizer rule that upgrades a scan from `Unsupported` to
+`Exact`/`Inexact` — and that removes the resulting redundant
+`SortExec` — is [`PushdownSort`](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pushdown_sort.rs). `PushdownSort`
+runs late, after `EnsureRequirements` has finalised the plan shape.
+It walks each `SortExec`, asks the child leaf via `try_pushdown_sort`
+which flavour the source can produce, and rewrites accordingly.
+
+## The `Exact` Path · Sort Elimination via Statistics
+
+<img src="/blog/images/sort-pushdown/phase1-file-reorder.svg" alt="File reorder: rearranging files within a partition by min/max statistics so the file list is in range order" width="100%" class="img-fluid" /><br/>
+*Figure: file reorder by per-file `min/max` puts the file list in range
+order without touching file contents.*
+
+DataFusion could already recognize the *exact* sortedness case (declared
+ordering + matching on-disk file list). The new capability is recognizing
+sortedness when the **file list is in the wrong order** on disk, using
+the min/max statistics that the Parquet writer already stored per row
+group. Implemented across two PRs on `PushdownSort`:
+[apache/datafusion#19064][#19064] (rule scaffolding), and
+[apache/datafusion#21182][#21182] (stats-based file reorder).
+
+[#19064]: https://github.com/apache/datafusion/pull/19064
+[#21182]: https://github.com/apache/datafusion/pull/21182
+
+For example, consider three files `a.parquet`, `b.parquet`,
+`c.parquet`. Each is internally sorted by `ts` and declares
+`WITH ORDER (ts ASC)`, but they were written by different jobs and end
+up listed alphabetically on disk (which does *not* match sort order).
+The old machinery has no way to prove global sortedness, so an
+`ORDER BY ts` query pays for a full external sort even though the
+underlying data is already sorted.
+
+`PushdownSort` fixes this in three steps at the file-scan node:
+
+1. **Sort the file list by per-file `min`** on the sort column.
+2. **Check adjacency**: does `file[i].max ≤ file[i+1].min` hold for
+ every adjacent pair? If yes, the sorted file list produces a globally
+ sorted stream.
+3. **Upgrade the source's ordering claim to `Exact`** and remove the
+ surrounding `SortExec`.
+
+<img src="/blog/images/sort-pushdown/phase2-stats-overlap.svg" alt="Detecting non-overlapping ranges via min/max statistics" width="100%" class="img-fluid" /><br/>
+*Figure: after reorder, the left case has non-overlapping ranges (safe
+to upgrade to `Exact`); the right case has overlaps (upgrade skipped,
+falls through to the `Inexact` path).*
+
+Two conservative bail-outs: (a) sort keys must be plain columns
+(`ORDER BY date_trunc('hour', ts)` doesn't qualify — no per-file min/max
+for the function output), and (b) sort columns must be null-free, so
+`NULLS FIRST`/`NULLS LAST` semantics are preserved across file
+boundaries. The overlap case falls through to the `Inexact` path
+covered later.
+
+### `BufferExec` · a subtle multi-partition side effect
+
+<img src="/blog/images/sort-pushdown/buffer-exec-stall.svg" alt="SPM stalls when SortExec is removed in multi-partition plans" width="100%" class="img-fluid" /><br/>
+*Figure: removing the per-partition `SortExec` leaves the top-of-plan
+merge (`SortPreservingMergeExec`) directly consuming raw I/O; a stall
+on any partition stalls the whole plan.*
+
+Removing the `SortExec` looked like a pure win, but the first
+multi-partition benchmarks showed something counter-intuitive: **some
+queries got slower**. The root cause is that the removed `SortExec`
+was doing two jobs — sorting *and* implicitly buffering. Each
+per-partition `SortExec` runs as its own task, greedily draining its
+source in the background; the top-of-plan `SortPreservingMergeExec`
+picks from those large in-memory buffers and never blocks on I/O in
+any single partition.
+
+Once the `SortExec` is deleted, the merge sits directly on the raw
+parquet streams. It's a lazy consumer — a k-way merge must see the
+head row from every input before deciding which to emit. A stall in
+*any one* partition now stalls the entire merge.
+
+<img src="/blog/images/sort-pushdown/buffer-exec.svg" alt="BufferExec replaces the deleted SortExec with a bounded streaming buffer per partition" width="100%" class="img-fluid" /><br/>
+*Figure: `BufferExec` is inserted where the `SortExec` used to live —
+same greedy per-partition prefill, but no blocking sort.*
+
+The fix is [`BufferExec`](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/buffer.rs): a bounded per-partition
+prefill buffer that plays the same "greedy parallel I/O driver" role
+the `SortExec` implicitly did. No sort, no blocking, and strictly
+less memory than the `SortExec` it replaces. The capacity is bounded
+(default 1 GB, configurable via
+[`sort_pushdown_buffer_capacity`](https://github.com/apache/datafusion/pull/21426)) and grows via the
+global memory pool, so it back-pressures the source instead of
+OOMing.
+
+### Benchmark: `sort_pushdown` suite
+
+<img src="/blog/images/sort-pushdown/benchmark.svg" alt="Sort pushdown benchmark: 2x-49x speedup across four queries" width="100%" class="img-fluid" /><br/>
+*Figure: `sort_pushdown` results (`--partitions 1`, release build). ASC
+queries with the file list reversed against sort-key ranges.*
+
+Numbers below are the [`sort_pushdown`](https://github.com/apache/datafusion/tree/main/benchmarks/queries/sort_pushdown) suite,
+`--partitions 1`, versus `main`:
+
+| Query | Before | After | Speedup |
+| ------------------------------------------- | -------:| -------:| -------: |
+| Q1 — `ORDER BY key` (full scan) | 259 ms | 122 ms | **2.1×** |
+| Q2 — `ORDER BY key LIMIT 100` | 80 ms | 3 ms | **27×** |
+| Q3 — `SELECT * ORDER BY key` | 700 ms | 313 ms | **2.2×** |
+| Q4 — `SELECT * ORDER BY key LIMIT 100` | 342 ms | 7 ms | **49×** |
+
+- **Full-scan queries (Q1, Q3)** save the cost of the sort itself
+ (~½ end-to-end latency for in-memory sorts).
+- **`LIMIT` queries (Q2, Q4)** benefit dramatically because deleting
+ the `SortExec` turns `LIMIT N` into a **static fetch** on the source —
+ the reader stops after N rows. A 342 ms full-file scan collapses
+ into a 7 ms K-row read.
+
+## The `Inexact` Path · Runtime Reorder for `TopK` and `DESC`
+
+Stats-based sort elimination handles the `Exact` upgrade — strong
+correctness, sort elimination — but only when the table has a
+declared `output_ordering` *and* the files are provably
+non-overlapping after sorting by min. Three classes of queries
+fall outside that window:
+
+* **Unsorted data** — no `WITH ORDER`, no parquet `sorting_columns`.
+ The `Exact` upgrade cannot fire because there is no ordering
+ claim to upgrade.
+* **Overlapping ranges** — files written by different ingestion
+ jobs share time windows. The `Exact` upgrade keeps the `SortExec`
+ because the global ordering can't be proven, even though the
+ files often do contain large stretches of in-order data.
+* **`ORDER BY ... DESC` on ASC-sorted data** — flipping iteration
+ at the row-group level emits "RGs descending × rows ascending",
+ close to the requested order but not strictly DESC, so the
+ `SortExec` has to stay for correctness.
+
+For all three, a full external `SortExec` is overkill. The parquet
+metadata is right there, and reading the *most-promising* data
+first lets `TopK`'s dynamic filter threshold tighten quickly so the
+rest gets pruned. Runtime reorder wires that up by generalising
+the `Inexact` path the rule introduced.
+
+### When Inexact fires
+
+<img src="/blog/images/sort-pushdown/pr21956-decision.svg" alt="try_pushdown_sort decision tree: Exact, Inexact, or Unsupported" width="100%" class="img-fluid" /><br/>
+*Figure: for each `SortExec`, the leaf source returns `Exact` (drop
+the sort), `Inexact` (bias the scan and keep the sort), or
+`Unsupported`.*
+
+The Inexact verdict fires when either of two independent signals is
+true:
+
+- **Stats-based reorder available**: the leading sort key is a plain
+ column in the file schema, so the scan can sort files and row
+ groups by `min(col)` from Parquet statistics.
+- **Reverse satisfies the request**: the source's declared ordering,
+ when reversed, satisfies what the query asks for. This uses
+ DataFusion's [equivalence-properties][ordering-analysis] reasoning
+ and covers function monotonicity (`ts DESC` declared, `date_trunc('day', ts) ASC`
+ requested), constants inferred from filters, and multi-column
+ composite orderings.
+
+### How the scan reorders data
+
+<img src="/blog/images/sort-pushdown/pr21956-runtime-pipeline.svg" alt="Runtime reorder pipeline: file reorder, RG reorder, then optional reverse" width="100%" class="img-fluid" /><br/>
+*Figure: the parquet opener applies file-level reorder → row-group-level
+reorder → optional iteration reverse.*
+
+The parquet opener applies up to three composable steps at query start:
+
+1. **File-level reorder** — across a shared work-stealing queue, the
+ file list is sorted by `min(col)`, so the most-promising file is
+ picked first across all partitions.
+2. **Row-group-level reorder** — once a file is opened, its row groups
+ are sorted by `min(col)`.
+3. **Iteration reverse** — flip row-group iteration order for `DESC`
+ requests (and for the reverse-satisfies-the-request cases above).
+
+### File-level early stop already works
+
+<img src="/blog/images/sort-pushdown/desc_walk_file.png" alt="Tier 1 file-level reorder with early stop via file_pruner" width="100%" class="img-fluid" /><br/>
+*Figure: after file reorder, low-value files at the tail of the queue
+are cut by the file-level pruner before they are ever opened — no
+metadata I/O.*
+
+Once files are ordered "most-promising first", `TopK`'s heap fills
+quickly and its dynamic filter threshold tightens. Low-value files at
+the tail of the queue are then checked against the live threshold
+by the [`FilePruner`](https://github.com/apache/datafusion/blob/main/datafusion/pruning/src/file_pruner.rs) before they are ever opened —
+never loading their footer, page index, or any data.
+
+### Row-group-level: the gap [#22450] fills
+
+<img src="/blog/images/sort-pushdown/desc_walk_rg.png" alt="Tier 2 RG-level reorder — filter column still read for every RG pre-#22450" width="100%" class="img-fluid" /><br/>
+*Figure: inside a file, the first row group tightens the threshold —
+subsequent row groups have their projection columns short-circuited,
+but the filter column still has to be read to discover that no rows
+qualify.*
+
+Inside a file, the story is almost identical — but with one gap.
+After the first row group fills the heap, subsequent row groups
+whose values can't beat the threshold evaluate to an empty
+`RowSelection`, and arrow-rs's reader short-circuits: no projection
+columns fetched, no decompress, no decode.
+
+However, **the filter column still gets read for every row group**,
+because the dynamic filter has to be evaluated row-by-row to
+*discover* that no rows survive. On a large file with many row
+groups, that's a meaningful tax — most of which is redundant, since
+metadata alone could have proven the row group unwinnable. Closing
+that gap is what [#22450] does.
+
+## #22450 · Runtime Row-Group Dynamic Pruning
+
+The merge that just landed — [apache/datafusion#22450][#22450] —
+re-checks the dynamic filter **at every row-group boundary** inside
+an open file, converts the live threshold into a fresh
+`PruningPredicate`, and physically removes any row group whose
+min/max can't possibly beat the threshold. The pruned row groups are
+**never decoded, not even on the filter column**.
+
+### Architecture · who drives the IO + decode loop
+
+<img src="/blog/images/sort-pushdown/arch_one_glance.png" alt="Three eras of who drives the parquet IO + decode loop" width="100%" class="img-fluid"/>
+
+The interesting backstory is that **DataFusion didn't actually own
+this loop until recently**. Three eras:
+
+* **Pre-[#20839]**: arrow-rs owned the I/O + decode loop as a black
+ box; DataFusion only called `.next()` and served byte ranges. The
+ row-group list was frozen at construction, so once the loop started,
+ no mid-stream decisions were possible.
+* **[#20839]**: the push-based parquet decoder moved the loop into
+ DataFusion. The capability to insert a decision mid-loop now
+ existed — but the loop went from `drain` straight to `drive`, with
+ no decision point.
+* **[#22450]**: adds the missing decision point. At every row-group
+ boundary, the loop pauses to ask the runtime pruner whether the
+ remaining row groups are still worth reading.
+
+### The loop, and the decision point [#22450] adds
+
+<img src="/blog/images/sort-pushdown/transition_anatomy.png" alt="transition() loop: drain, decide, drive — Step 2 is the #22450 addition" width="100%" class="img-fluid" /><br/>
+*Figure: the decoder loop has three steps. Step 2 (DECIDE) is what
+[#22450] adds — it only fires at row-group boundaries.*
+
+The loop body reads: **drain** the current row group's batches until
+it's exhausted; **decide** at the boundary whether any of the
+remaining row groups can be dropped based on the live threshold; then
+**drive** the decoder into the next row group and repeat. Inside a
+row group, only drain and drive run — no decision point.
+
+<img src="/blog/images/sort-pushdown/pruner_loop.png" alt="RowGroupPruner: watch (cheap), rebuild (expensive, only if changed), prune (cheap)" width="100%" class="img-fluid" /><br/>
+*Figure: the pruner has a cheap "check if the filter changed" step, a
+moderately expensive "rebuild the predicate if so" step, and a cheap
+"apply the predicate to remaining row groups" step.*
+
+The pruner is designed so the expensive work only fires when it can
+possibly help: a cheap epoch check tells it whether the dynamic filter
+has actually changed since last time, and only then does it rebuild
+the pruning predicate. The predicate is then applied to remaining
+row groups' min/max statistics — pure metadata comparison, no I/O.
+Errors always fall back to "keep the row group" — a flaky pruner
+never drops live data.
+
+### Cascading prune · how the heap eats row groups
+
+<img src="/blog/images/sort-pushdown/rg_cascade.png" alt="Cascading prune: one row group fills the heap, threshold snaps, all subsequent row groups are pruned in a single pass" width="100%" class="img-fluid" /><br/>
+*Figure: for `ORDER BY x DESC LIMIT 10`, opening the first row group
+(values [90..100)) is enough to fill the heap; at the next boundary,
+every remaining row group with `max < 90` is pruned in one pass.*
+
+The savings compound because the threshold moves in **steps**, not
+smoothly. For `ORDER BY x DESC LIMIT 10` on a 10-row-group file
+where reorder puts high-value row groups first:
+
+1. RG 9 (values `[90..100)`) opens. One row group is enough to fill
+ the heap of size 10 — the threshold jumps into RG 9's range (≥ 90).
+2. At the next row-group boundary, the pruner sees that all of RG 8
+ through RG 0 have `max < 90` and drops them in one pass.
+3. Bytes for those nine row groups were **never fetched** — not
+ projection columns, not the filter column. Full I/O + decompress +
+ decode all skipped.
+
+This is the unconditional value of [#22450]: when reorder lines up
+disjoint per-RG ranges (the common case for time-series or
+partition-key sorts), a single row group can cascade-eliminate every
+remaining row group at the next boundary.
+
+## Three-Layer Pruning · file + RG + row, stacked
+
+<img src="/blog/images/sort-pushdown/pruning_stack.png" alt="Three-layer pruning: file-level, RG-level, row-level, all driven by the same TopK dynamic filter" width="100%" class="img-fluid"/>
+
+A common question at this point: "if [#22450] prunes whole row
Review Comment:
Done — reframed as an intro to the multi-layer pruning strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org