You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/08/16 11:30:17 UTC
[PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
alamb opened a new pull request, #203:
URL: https://github.com/apache/datafusion-site/pull/203
## Which issue does this PR close?
- Related to https://github.com/apache/datafusion/issues/24216
- Part of https://github.com/apache/datafusion/issues/22393
## Rationale for this change
Draft blog post announcing the DataFusion 55.0.0 release, following the model of the [54.0.0 post](https://github.com/apache/datafusion-site/pull/197).
**This is an early draft and is NOT ready for review yet.** Known TODOs:
- [ ] Generate the ClickBench performance-over-time chart for 55.0.0 (`content/images/datafusion-55.0.0/performance_over_time_clickbench.png`)
- [ ] Confirm the final release/publish date (currently 2026-08-16) and rename the file to match
- [ ] Verify feature descriptions, credits, and reviewer attributions with the contributors involved
- [ ] Double-check which late-merged PRs actually made the branch-55 cut
Content highlights (drawn from the [55.0.0 release ticket](https://github.com/apache/datafusion/issues/22393) feature list):
- Range partitioning (highlighted as a key new feature per discussion on the blog ticket)
- MERGE INTO planner + TableProvider hook
- Per-partition TopK for window functions (`ROW_NUMBER`/`RANK`)
- Sort pushdown phase 4: runtime row-group pruning for TopK
- Parquet virtual columns (`file_row_index`, `input_file_name`)
- `unnest_outer`, pluggable spill backends, UDF strictness metadata
- Extensibility APIs for distributed engines (FFI_QueryPlanner, dynamic filter APIs, proto hooks)
- Aggregation, function, planner, and scan performance improvements
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3854893211
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
Review Comment:
NGL, this comes across as a bit self congratulatory. Obviously: it is great! But could soften the language a bit.
```suggestion
the board rather than further optimizing our already very satisfying ClickBench numbers
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both with a breaking for the `MergeIntoOp` construction. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3853695271
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
(I will trim the section down a bit but wanted to capture the content first)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3797587969
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
maybe a quick example or link to documented example to show how split points are used 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
@alamb Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both in the planned 55.1 release with a breaking for the `MergeIntoOp` construction. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3862820913
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already very satisfying ClickBench numbers
Review Comment:
nit: "already ... already" repeats with the next line ("DataFusion is already the fastest"), maybe drop the first one:
```suggestion
the board rather than further optimizing our very satisfying ClickBench numbers
```
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
+Thanks to [@wirybeaver] for implementing this feature, with reviews from
+[@alamb] and [@kosiew].
+
+
+### Pluggable Spill Backends
+
+DataFusion spills to disk when a query exceeds its memory budget, but the spill
+infrastructure was previously hardwired to OS-level temporary files. DataFusion
+55 introduces a pluggable [`SpillFile`][SpillFile] trait and
+[`TempFileFactory`][TempFileFactory] ([#21882],
+[#22230]) so hosts can route spill data through their own storage layers — for
+example, extensions like ParadeDB can now integrate spilling into the
+Postgres buffer pool. Implement [`TempFileFactory`][TempFileFactory] and install
+it on the [`RuntimeEnv`][RuntimeEnv]:
+
+```rust
+let runtime = RuntimeEnvBuilder::new()
+ .with_disk_manager_builder(
+ // register a custom TempFileFactory
+ DiskManagerBuilder::default()
+ .with_temp_file_factory(Arc::new(MyTempFileFactory::new())),
+ )
+ .build_arc()?;
+let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
+```
+
+See the [`object_store_spill.rs`][object_store_spill.rs] example for a complete implementation that
+spills to an [`ObjectStore`][ObjectStore] such as S3. Thanks to [@pantShrey] for this work,
+with reviews from [@alamb].
+
+### Extensibility for Distributed Engines
+
+Several new APIs make it easier to build distributed systems such as
+[datafusion-distributed], [DataFusion Ballista], and [DataFusion Python] on
+top of DataFusion:
+
+- **Dynamic filter propagation across network boundaries**: new
+ [`ExecutionPlan::apply_expressions`][ExecutionPlan::apply_expressions] and
+ [`ExecutionPlan::dynamic_expressions_produced`][ExecutionPlan::dynamic_expressions_produced] methods let engines discover
+ which plan nodes produce dynamic filters and re-wire them across stage
+ boundaries ([#24018], [#24068]). Thanks to [@jayshrivastava].
+- **`FFI_QueryPlanner`**: foreign libraries can now provide a custom query
+ planner over the FFI boundary — for example, connecting a distributed
+ planner to a [`SessionContext`][SessionContext] in Python ([#24028]). Thanks to [@timsaucer].
+- **Self-serializing execution plans**: built-in [`ExecutionPlan`][ExecutionPlan]s were ported
+ to per-type [`try_to_proto`][try_to_proto] / [`try_from_proto`][try_from_proto] hooks ([#23494]), putting built-in and
+ third-party plans on the same code path. Thanks to [@adriangb].
+- **Window accumulator state access**: [`BoundedWindowAggExec`][BoundedWindowAggExec] can now expose
+ finalized accumulator state to an observer callback, enabling incremental /
+ prefix-scan use cases ([#24035]; see [how it is used in Ballista]). Thanks to
+ [@avantgardnerio], with reviews from [@alamb] and [@timsaucer].
+
+[datafusion-distributed]: https://github.com/datafusion-contrib/datafusion-distributed
+[DataFusion Ballista]: https://datafusion.apache.org/ballista/
+[DataFusion Python]: https://datafusion.apache.org/python/
+[ExecutionPlan::apply_expressions]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html#tymethod.apply_expressions
+[ExecutionPlan::dynamic_expressions_produced]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html#method.dynamic_expressions_produced
+[SessionContext]: https://docs.rs/datafusion/55.0.0/datafusion/execution/context/struct.SessionContext.html
+[ExecutionPlan]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/trait.ExecutionPlan.html
+[try_to_proto]: https://docs.rs/datafusion-physical-plan/55.0.0/datafusion_physical_plan/filter/struct.FilterExec.html#method.try_to_proto
+[try_from_proto]: https://docs.rs/datafusion-physical-plan/55.0.0/datafusion_physical_plan/filter/struct.FilterExec.html#method.try_from_proto
+[BoundedWindowAggExec]: https://docs.rs/datafusion/55.0.0/datafusion/physical_plan/windows/struct.BoundedWindowAggExec.html
+[how it is used in Ballista]: https://github.com/apache/datafusion-ballista/pull/2211
+
+### `EXPLAIN` Improvements
+
+DataFusion 55 adds a Postgres-style `EXPLAIN (...)` option list ([#21768]) and
+a `pgjson` output format for [`EXPLAIN ANALYZE`][EXPLAIN ANALYZE] ([#21767]), making plan output
+easier to consume with existing Postgres tooling such as plan visualizers.
+
+[EXPLAIN ANALYZE]: https://datafusion.apache.org/user-guide/explain-usage.html#execution-counters-explain-analyze
+
+`EXPLAIN ANALYZE` can produce many metrics, and you can narrow them down
+explicitly with the `METRICS` option. For example, to see only row counts for
+each plan node, use this:
+
+```sql
+> EXPLAIN (ANALYZE, METRICS 'rows', LEVEL summary)
+ SELECT region_id, sum(amount) FROM orders GROUP BY region_id;
+
+AggregateExec: mode=Single, gby=[region_id@0 as region_id], aggr=[sum(orders.amount)], metrics=[output_rows=5]
+ DataSourceExec: file_groups={1 group: [[orders.parquet]]}, projection=[region_id, amount], file_type=parquet, metrics=[output_rows=100.0 K, row_groups_pruned_statistics=1 total → 1 matched, scan_efficiency_ratio=7.83% (28.30 K/361.5 K), ...]
+```
+
+Adding `FORMAT pgjson` renders the same physical plan and its live metrics as
+Postgres-compatible JSON, which can be pasted straight into plan visualizers
+such as [Dalibo]:
+
+```sql
+> EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows', LEVEL summary)
+ SELECT region_id, sum(amount) FROM orders GROUP BY region_id;
+```
+
+```json
+[
+ {
+ "Plan": {
+ "Node Type": "AggregateExec",
+ "Actual Rows": 5,
+ "Plans": [
+ { "Node Type": "DataSourceExec", "Actual Rows": 100000, ... }
+ ]
+ }
+ }
+]
+```
+
+See the [EXPLAIN usage guide] for the full option list. Thanks to [@adriangb]
+for this work.
+
+[Dalibo]: https://explain.dalibo.com/
+[EXPLAIN usage guide]: https://datafusion.apache.org/user-guide/explain-usage.html
+
+### New Functions
+
+**SQL and Scalar Functions**:
+DataFusion 55 adds new array math functions [`array_scale`][array_scale], [`array_add`][array_add],
+[`array_subtract`][array_subtract], [`array_sum`][array_sum], and [`array_avg`][array_avg], plus the higher-order
+[`array_first`][array_first] function. It also includes the new
+[`unnest_outer`][unnest_outer] function ([#22100]), which preserves
+empty inputs as `NULL`s rather than dropping them, and
+[`approx_distinct`][approx_distinct] now supports more types.
+Thanks to [@crm26], [@SubhamSinghal], [@EdsonPetry], [@athlcode], and
+[@mkleen] for these contributions.
+
+**Spark-Compatible Functions**:
+The [datafusion-spark crate] gains new or improved Spark-compatible functions,
+including [`hypot`][hypot], [`atan2`][atan2], [`weekday`][weekday], [`monthname`][monthname], and [`concat_ws`][concat_ws] with array
+support, plus a new Spark SQL parser dialect config ([#22529]).
+Thanks to the contributors who drove this work, especially
+[@KarpagamKarthikeyan], [@sjhddh], [@JeelRajodiya], [@davidlghellin], and
+[@kumarUjjawal].
+
+## Upgrade Guide and Changelog 📖
+
+Upgrading to 55.0.0 should be straightforward for most users, though there are
+some breaking changes. See the [Upgrade Guide] for details and
+migration snippets, and the [changelog] for the full list of changes.
+
+## About DataFusion
+
+[Apache DataFusion] is an extensible query engine, written in [Rust], that uses
+[Apache Arrow] as its in-memory format. DataFusion is used by developers to
+create new, fast, data-centric systems such as databases, dataframe libraries,
+and machine learning and streaming applications. While [DataFusion's primary
+design goal] is to accelerate the creation of other data-centric systems, it
+provides a reasonable experience directly out of the box as a [dataframe
+library], [Python library], and [command-line SQL tool].
+
+DataFusion's core thesis is that, as a community, together we can build much
+more advanced technology than any of us as individuals or companies could build
+alone. Without DataFusion, highly performant vectorized query engines would
+remain the domain of a few large companies and world-class research
+institutions. With DataFusion, we can all build on top of a shared foundation
+and focus on what makes our projects unique.
+
+## How to Get Involved
+
+DataFusion is not a project built or driven by a single person, company, or
+foundation. Rather, our community of users and contributors works together to
+build a shared technology that none of us could have built alone.
+
+If you are interested in joining us, we would love to have you. You can try out
+DataFusion on some of your own data and projects and let us know how it goes,
+contribute suggestions, documentation, bug reports, or a PR with documentation,
+tests, or code. A list of open issues suitable for beginners is [here], and you
+can find out how to reach us on the [communication doc].
+
+## Appendix: ClickBench Results
+
+We try not to get too excited by benchmarks, though it is hard not to get caught up
+in Benchmaxxing. As noted
+above, ClickBench covers only a tiny fraction of what our users actually do, and
+reads local files rather than object storage. Even so, DataFusion now sits at
+the top of the ClickBench leaderboard for processing partitioned Parquet files,
+as measured by ClickBench's combined metric. Results vary
+slightly by engine and instance type, but DataFusion matches other
+state-of-the-art engines on local files and often significantly exceeds them on
+object storage. Plenty left to optimize, of course.
+
+
+<img
+src="/blog/images/datafusion-55.0.0/clickbench.c7a.metal-48xlarge.2026_08_24.png"
+width="100%"
+class="img-fluid"
+alt="DataFusion ClickBench performance on c7a.metal-48xlarge"
+/>
+**Figure 2**: [ClickBench results for c7a.metal-48xlarge as of 2026-08-24]; DataFusion is the fastest engine for
+processing partitioned Parquet files, by ClickBench's combined metric. See the [ClickBench results page] for the latest results.
+
+[ClickBench results for c7a.metal-48xlarge as of 2026-08-24]: https://benchmark.clickhouse.com/#system=+hqa|curp|ti%20rud|Distt|kBP%20t|rsoP%20t|Sili|feeid|traan|Tutt&type=-&machine=+ae-&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-
+[ClickBench results page]: https://benchmark.clickhouse.com/
+
+<img
+src="/blog/images/datafusion-55.0.0/clickbench.c6a.4xlarge.2026_08_24.png"
+width="100%"
+class="img-fluid"
+alt="DataFusion ClickBench performance on c6a.4xlarge"
+/>
+**Figure 3**: [ClickBench results for c6a.4xlarge as of 2026-08-24]; DataFusion is the second
+fastest engine for processing partitioned Parquet files (combined metric) on this VM type. See the [ClickBench results page] for the latest results.
+
+[ClickBench results for c6a.4xlarge as of 2026-08-24]: https://benchmark.clickhouse.com/#system=+hqa|curp|ti%20rud|Distt|kBP%20t|rsoP%20t|Sili|feeid|traan|Tutt&type=-&machine=+ca4e&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-
+
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="DataFusion Performance over time"
+/>
+
+**Figure 4**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+## Appendix: Full List of Performance Improvements
+
+The tables below list the performance improvements in this release along with a
+representative measurement for each. Results marked *(micro)* come from
+Criterion microbenchmarks and are not expected to translate directly into
+end-to-end query speedups. Speedups are reported as ratios of old to new
+runtime: "n% faster" means the old runtime was (100+n)% of the new, and
+speedups of 2x or more are reported as a multiple.
+
+### Sort / TopK
+
+| Improvement | Issue / PR | Representative Result |
+|------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
+| Runtime row-group pruning for TopK | [#23036](https://github.com/apache/datafusion/issues/23036), [#22450](https://github.com/apache/datafusion/pull/22450) | [4.2x faster on `topk_tpch` Q8; 5 of 11 queries 3.6-4.2x faster](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) |
+| Faster `SortPreservingMerge` tie-breaker | [#23107](https://github.com/apache/datafusion/pull/23107) | [8% faster on `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) |
+
+### Window Functions
+
+| Improvement | Issue / PR | Representative Result |
+|-----------------------------------------------|-------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
+| `LEAD` / `LAG` with `IGNORE NULLS` | [#23711](https://github.com/apache/datafusion/pull/23711) | [21.7x faster for `List`, 10.5x for `Utf8View`](https://github.com/apache/datafusion/pull/23711) *(micro)* |
+| Sliding-window `MIN` / `MAX` monotonic deques | [#23827](https://github.com/apache/datafusion/pull/23827) | [up to 3.5x faster (strings; ~2x for numerics)](https://github.com/apache/datafusion/pull/23827#issuecomment-5067789139) *(micro)* |
+| Skip fully calculated window partitions | [#24127](https://github.com/apache/datafusion/pull/24127) | [49% faster with 32,768 sparse partitions: 161ms → 108ms](https://github.com/apache/datafusion/pull/24127) |
+| Skip re-slicing quiet window partitions | [#24047](https://github.com/apache/datafusion/pull/24047) | [26% faster with 32,768 sparse partitions: 209ms → 166ms](https://github.com/apache/datafusion/pull/24047) |
+
+### Aggregation
+
+| Improvement | Issue / PR | Representative Result |
+|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `approx_distinct` with many groups | [#22768](https://github.com/apache/datafusion/pull/22768) | [101x faster: 1723ms → 17ms (Int64, 50K groups)](https://github.com/apache/datafusion/pull/22768#issuecomment-4627539820) |
+| `array_agg(DISTINCT ...)` | [#23716](https://github.com/apache/datafusion/pull/23716) | [4.0x faster at high cardinality](https://github.com/apache/datafusion/pull/23716) *(micro)* |
+| `percentile_cont` / `median` | [#23954](https://github.com/apache/datafusion/pull/23954) | [2.7x faster: 247µs → 92µs (`median`, window=256)](https://github.com/apache/datafusion/pull/23954) *(micro)* |
+| Multi-column `GROUP BY` type coverage | [#22715](https://github.com/apache/datafusion/issues/22715), [#23523](https://github.com/apache/datafusion/pull/23523) | [46% less memory for mixed-schema keys: 1096KB → 594KB](https://github.com/apache/datafusion/pull/23523) |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [#23645](https://github.com/apache/datafusion/issues/23645), [#23646](https://github.com/apache/datafusion/pull/23646) | [~5% faster: 1.13s → 1.07s grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), addressing a [reported out-of-memory crash](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) |
+| Semi / anti join index alignment | [#22794](https://github.com/apache/datafusion/pull/22794) | [TPC-DS Q15 16% faster](https://github.com/apache/datafusion/pull/22794#issuecomment-4640131159) |
+
+### Expressions and Functions
+
+| Improvement | Issue / PR | Representative Result |
+|--------------------------------------------------------------------------| --- |----------------------------------------------------------------------------------------------------------------------------|
+| Faster `IN` list evaluation | [#19241](https://github.com/apache/datafusion/issues/19241), [#23014](https://github.com/apache/datafusion/pull/23014) | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014) *(micro)* |
+| Faster `IN` list (`UInt8` bitmap) | [#23011](https://github.com/apache/datafusion/pull/23011) | [9.7x faster: 38.4µs → 4.0µs](https://github.com/apache/datafusion/pull/23011) *(micro)* |
+| Faster `IN` list (`Int8` / `Int16`) | [#23299](https://github.com/apache/datafusion/pull/23299) | [4.4x faster for `i16`, 4-element list](https://github.com/apache/datafusion/pull/23299#issuecomment-4875954295) *(micro)* |
+| Preserve dictionary encoding for string functions | [#23930](https://github.com/apache/datafusion/pull/23930) | [360x faster `initcap` on cardinality-10 dictionaries](https://github.com/apache/datafusion/pull/23930) *(micro)* |
+| Preserve dictionary encoding for trim family | [#24100](https://github.com/apache/datafusion/pull/24100) | [151x faster `ltrim` on cardinality-10 dictionaries](https://github.com/apache/datafusion/pull/24100) *(micro)* |
+| Preserve dictionary encoding for (`ascii`, `bit_length`, `octet_length`) | [#23743](https://github.com/apache/datafusion/pull/23743) | [28.7x faster `ascii`](https://github.com/apache/datafusion/pull/23743) *(micro)* |
+| `find_in_set` | [#23460](https://github.com/apache/datafusion/pull/23460) | [24x faster: 1.25ms → 52µs](https://github.com/apache/datafusion/pull/23460) *(micro)* |
+| `array_has` with array needle | [#23337](https://github.com/apache/datafusion/pull/23337) | [15.1x faster; join query 0.95s → 0.059s](https://github.com/apache/datafusion/pull/23337) |
+| `trunc` with scalar precision | [#23593](https://github.com/apache/datafusion/pull/23593) | [12.9x faster: 9.5µs → 735ns](https://github.com/apache/datafusion/pull/23593) *(micro)* |
+| `encode` / hex | [#23456](https://github.com/apache/datafusion/pull/23456) | [5.1x faster: 200µs → 39.5µs](https://github.com/apache/datafusion/pull/23456) *(micro)* |
+| `arrays_zip` perfect-list fast path | [#22285](https://github.com/apache/datafusion/pull/22285) | [4.2x faster](https://github.com/apache/datafusion/pull/22285#issuecomment-4537748133) *(micro)* |
+| `overlay` | [#22182](https://github.com/apache/datafusion/pull/22182) | [5.3x faster on high-null input: 401µs → 75µs](https://github.com/apache/datafusion/pull/22182) *(micro)* |
+| `translate` | [#22171](https://github.com/apache/datafusion/pull/22171) | [4.4x faster: 265µs → 60µs](https://github.com/apache/datafusion/pull/22171) *(micro)* |
+| `date_trunc` | [#23542](https://github.com/apache/datafusion/pull/23542) | [4.2x faster for `week`: 23.3µs → 5.6µs](https://github.com/apache/datafusion/pull/23542) *(micro)* |
+| `replace` | [#23589](https://github.com/apache/datafusion/pull/23589) | [2.5x faster: 228µs → 91µs](https://github.com/apache/datafusion/pull/23589) *(micro)* |
+| `regexp_instr` | [#23540](https://github.com/apache/datafusion/pull/23540) | [90% faster: 49.2µs → 26.0µs](https://github.com/apache/datafusion/pull/23540) *(micro)* |
+| `left` / `right` | [#23762](https://github.com/apache/datafusion/pull/23762) | [81% faster on `string_view` long results: 94µs → 52µs](https://github.com/apache/datafusion/pull/23762) *(micro)* |
+| `round` | [#23471](https://github.com/apache/datafusion/pull/23471) | [79% faster: 1486ns → 831ns](https://github.com/apache/datafusion/pull/23471) *(micro)* |
+| `regexp_match` with literal pattern | [#23547](https://github.com/apache/datafusion/pull/23547) | [59% faster with literal pattern and flags: 280µs → 176µs](https://github.com/apache/datafusion/pull/23547) *(micro)* |
+| `date_part` `isodow` | [#23491](https://github.com/apache/datafusion/pull/23491) | [reported 38% improvement](https://github.com/apache/datafusion/pull/23491#issuecomment-4962484527) *(micro)* |
+
+### Planning
+
+| Improvement | Issue / PR | Representative Result |
+| --- | --- | --- |
+| Collapse chained projections | [#22389](https://github.com/apache/datafusion/pull/22389) | [4.0x faster planning: 623ms → 155ms](https://github.com/apache/datafusion/pull/22389) |
+| Skip subquery traversal, rewrite in place | [#22298](https://github.com/apache/datafusion/pull/22298) | [29% faster TPC-DS optimization: 220ms → 170ms](https://github.com/apache/datafusion/pull/22298) |
+| Don't re-inline CSE'd expensive expressions | [#23459](https://github.com/apache/datafusion/pull/23459) | [67% faster on repeated `power(a, 2)`](https://github.com/apache/datafusion/pull/23459) |
+| Skip `ensure_distribution` rebuild for unchanged children | [#22521](https://github.com/apache/datafusion/pull/22521) | [2.9x faster per call: 171µs → 59µs](https://github.com/apache/datafusion/pull/22521) |
+| Unified `EnsureRequirements` pass | [#21976](https://github.com/apache/datafusion/pull/21976) | [TPC-H: 8 queries faster, 0 slower](https://github.com/apache/datafusion/pull/21976#issuecomment-4521276189) |
+| Predicate reordering heuristic | [#22343](https://github.com/apache/datafusion/pull/22343) | [ClickBench Q21 10-13% faster](https://github.com/apache/datafusion/pull/22343#issuecomment-4483681627) |
+
+### Scan / IO
+
+| Improvement | Issue / PR | Representative Result |
+| --- | --- | --- |
+| Prune unread Parquet leaves for nested columns | [#24090](https://github.com/apache/datafusion/pull/24090) | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) |
+| Fewer object store requests for CSV | [#22962](https://github.com/apache/datafusion/pull/22962) | [70% faster on TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) |
+| Lower `repartition_file_min_size` to 1 MiB | [#22439](https://github.com/apache/datafusion/pull/22439) | [TPC-H Q22 68% faster](https://github.com/apache/datafusion/pull/22439#issuecomment-4511995613) |
+| Skip page index load when the file has none | [#24149](https://github.com/apache/datafusion/issues/24149), [#24150](https://github.com/apache/datafusion/pull/24150) | [ClickBench (single file) Q1 38% faster](https://github.com/apache/datafusion/pull/24150#issuecomment-5226213508) |
+
+
+[Apache DataFusion]: https://datafusion.apache.org/
+[Rust]: https://www.rust-lang.org/
+[Apache Arrow]: https://arrow.apache.org
+[DataFusion's primary design goal]: https://datafusion.apache.org/user-guide/introduction.html#project-goals
+[dataframe library]: https://datafusion.apache.org/user-guide/dataframe.html
+[Python library]: https://datafusion.apache.org/python/
+[command-line SQL tool]: https://datafusion.apache.org/user-guide/cli/
+[Upgrade Guide]: https://datafusion.apache.org/library-user-guide/upgrading/55.0.0.html
+[here]: https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
Review Comment:
nit: this still points at the old `arrow-datafusion` repo name (it redirects, but we can use the current name):
```suggestion
[here]: https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
@alamb Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both with a breaking for the `MergeIntoOp` construction. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3879518378
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,721 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
Review Comment:
🤔 the date will be a few days early, but I guess that is ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3832696638
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
sweet I will take time this weekend to help with edits here and in general 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3855969307
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,718 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
+Thanks to [@wirybeaver] for implementing this feature, with reviews from
+[@alamb] and [@kosiew].
+
+
+### Pluggable Spill Backends
+
+DataFusion spills to disk when a query exceeds its memory budget, but the spill
+infrastructure was previously hardwired to OS-level temporary files. DataFusion
+55 introduces a pluggable [`SpillFile`][SpillFile] trait and
+[`TempFileFactory`][TempFileFactory] ([#21882],
+[#22230]) so hosts can route spill data through their own storage layers — for
+example, extensions like ParadeDB can now integrate spilling into the
+Postgres buffer pool. Implement [`TempFileFactory`][TempFileFactory] and install
+it on the [`RuntimeEnv`][RuntimeEnv]:
+
+```rust
+let runtime = RuntimeEnvBuilder::new()
+ .with_disk_manager_builder(
+ // register a custom TempFileFactory
+ DiskManagerBuilder::default()
+ .with_temp_file_factory(Arc::new(MyTempFileFactory::new())),
+ )
+ .build_arc()?;
+let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
+```
+
+See the [`object_store_spill.rs`][object_store_spill.rs] example for a complete implementation that
+spills to an [`ObjectStore`][ObjectStore] such as S3. Thanks to [@pantShrey] for this work,
+with reviews from [@alamb].
+
+### Extensibility for Distributed Engines
+
+Several new APIs make it easier to build distributed systems such as
+[datafusion-distributed], [DataFusion Ballista], and [DataFusion Python] on
+top of DataFusion:
+
+- **Dynamic filter propagation across network boundaries**: new
+ [`ExecutionPlan::apply_expressions`][ExecutionPlan::apply_expressions] and
+ [`ExecutionPlan::dynamic_expressions_produced`][ExecutionPlan::dynamic_expressions_produced] methods let engines discover
+ which plan nodes produce dynamic filters and re-wire them across stage
+ boundaries ([#24018], [#24068]). Thanks to [@jayshrivastava].
+- **`FFI_QueryPlanner`**: foreign libraries can now provide a custom query
+ planner over the FFI boundary — for example, connecting a distributed
+ planner to a [`SessionContext`][SessionContext] in Python ([#24028]). Thanks to [@timsaucer].
+- **Self-serializing execution plans**: built-in [`ExecutionPlan`][ExecutionPlan]s were ported
+ to per-type [`try_to_proto`][try_to_proto] / [`try_from_proto`][try_from_proto] hooks ([#23494]), putting built-in and
+ third-party plans on the same code path. Thanks to [@adriangb].
+- **Window accumulator state access**: [`BoundedWindowAggExec`][BoundedWindowAggExec] can now expose
+ finalized accumulator state to an observer callback, enabling incremental /
+ prefix-scan use cases ([#24035]). Thanks to [@avantgardnerio], with reviews
Review Comment:
Perhaps a link to [how it is used](https://github.com/apache/datafusion-ballista/pull/2211), for the curious? FYI @gabotechs , JIC you are thinking about such things for datafusion-distributed. @gene-bordegaray too, because it's a use-case for dynamic range partitioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
@alamb Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both in the planned 56.0 release with a breaking for the `MergeIntoOp` construction. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3853692976
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
Thank you @gene-bordegaray -- the content about range partitioning was great, but I think a little too detailed for this release announcement. I filed a ticket about writing a self contained blog post
- https://github.com/apache/datafusion/issues/24667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3857634948
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
Review Comment:
🙇 Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3857590359
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
Review Comment:
Agreed. I will update.
As a bonus, see if you can spot the humble brag easter egg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3857607896
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,718 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
+Thanks to [@wirybeaver] for implementing this feature, with reviews from
+[@alamb] and [@kosiew].
+
+
+### Pluggable Spill Backends
+
+DataFusion spills to disk when a query exceeds its memory budget, but the spill
+infrastructure was previously hardwired to OS-level temporary files. DataFusion
+55 introduces a pluggable [`SpillFile`][SpillFile] trait and
+[`TempFileFactory`][TempFileFactory] ([#21882],
+[#22230]) so hosts can route spill data through their own storage layers — for
+example, extensions like ParadeDB can now integrate spilling into the
+Postgres buffer pool. Implement [`TempFileFactory`][TempFileFactory] and install
+it on the [`RuntimeEnv`][RuntimeEnv]:
+
+```rust
+let runtime = RuntimeEnvBuilder::new()
+ .with_disk_manager_builder(
+ // register a custom TempFileFactory
+ DiskManagerBuilder::default()
+ .with_temp_file_factory(Arc::new(MyTempFileFactory::new())),
+ )
+ .build_arc()?;
+let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
+```
+
+See the [`object_store_spill.rs`][object_store_spill.rs] example for a complete implementation that
+spills to an [`ObjectStore`][ObjectStore] such as S3. Thanks to [@pantShrey] for this work,
+with reviews from [@alamb].
+
+### Extensibility for Distributed Engines
+
+Several new APIs make it easier to build distributed systems such as
+[datafusion-distributed], [DataFusion Ballista], and [DataFusion Python] on
+top of DataFusion:
+
+- **Dynamic filter propagation across network boundaries**: new
+ [`ExecutionPlan::apply_expressions`][ExecutionPlan::apply_expressions] and
+ [`ExecutionPlan::dynamic_expressions_produced`][ExecutionPlan::dynamic_expressions_produced] methods let engines discover
+ which plan nodes produce dynamic filters and re-wire them across stage
+ boundaries ([#24018], [#24068]). Thanks to [@jayshrivastava].
+- **`FFI_QueryPlanner`**: foreign libraries can now provide a custom query
+ planner over the FFI boundary — for example, connecting a distributed
+ planner to a [`SessionContext`][SessionContext] in Python ([#24028]). Thanks to [@timsaucer].
+- **Self-serializing execution plans**: built-in [`ExecutionPlan`][ExecutionPlan]s were ported
+ to per-type [`try_to_proto`][try_to_proto] / [`try_from_proto`][try_from_proto] hooks ([#23494]), putting built-in and
+ third-party plans on the same code path. Thanks to [@adriangb].
+- **Window accumulator state access**: [`BoundedWindowAggExec`][BoundedWindowAggExec] can now expose
+ finalized accumulator state to an observer callback, enabling incremental /
+ prefix-scan use cases ([#24035]). Thanks to [@avantgardnerio], with reviews
Review Comment:
Added in 024399b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3861838922
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
Review Comment:
🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both with a breaking API change. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5450827552
ok, here we go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3854730383
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
Review Comment:
Since we didn't really improve ClickBench performance on this release I focused the blog on other performance improvements instead
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
Review Comment:
@gene-bordegaray here is a slimmed down version of the Range partitioning intro -- let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3872279640
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
Thanks @wirybeaver that is a good point -- I added a note in 00cc646 that we plan more MERGE INTO work in the following releases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5432518157
I'll plan to merge this on Friday unless anyone else woul like more time to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3797591508
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
I can of course make this change, just throwing idea out there first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #203:
URL: https://github.com/apache/datafusion-site/pull/203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5413238359
I think you should also mention that you can now opt in for better ordering equivalence (for example custom cast)
- https://github.com/apache/datafusion/pull/23807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3861835162
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,714 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our already great ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
Review Comment:
ya think this is good summary, too much to fit into a non dedicated post 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [NOT READY FOR REVIEW] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3832643295
##########
content/blog/2026-08-16-datafusion-55.0.0.md:
##########
@@ -0,0 +1,517 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-16
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post highlights
+some of the major improvements since [DataFusion 54.0.0]. Notable additions
+include range partitioning, `MERGE INTO` support, per-partition TopK for window
+functions, and runtime row-group pruning for TopK queries, alongside significant
+aggregation, function, and planning performance improvements. The complete list
+of changes is available in the [changelog]. This release represents roughly 9
+weeks of development and 877 commits. Thanks to the [175 contributors]
+(a new record!) for making it possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+## Performance Improvements 🚀
+
+<!-- TODO: generate updated performance chart for 55.0.0 and place it in
+ content/images/datafusion-55.0.0/performance_over_time_clickbench.png -->
+<img
+src="/blog/images/datafusion-55.0.0/performance_over_time_clickbench.png"
+width="100%"
+class="img-fluid"
+alt="Performance over time"
+/>
+
+**Figure 1**: Average and median normalized execution times for DataFusion 55.0.0 on ClickBench queries, compared to previous releases.
+Query times are normalized using the ClickBench definition. See the
+[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/)
+for more details.
+
+We continue to make significant performance improvements in DataFusion, as
+explained below. This release skips more work at runtime using statistics and
+dynamic filters, and makes window functions, aggregation, and many built-in
+functions faster.
+
+### Sort Pushdown: Runtime Row-Group Pruning for TopK Queries
+
+The multi-release [Sort Pushdown effort] makes `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries on Parquet skip work end-to-end: skip the
+sort, skip row groups via min/max statistics, and skip rows via dynamic filters.
+DataFusion 55 lands the next phase: as a TopK query runs and its dynamic filter
+threshold tightens, the Parquet reader now re-evaluates the threshold against
+the remaining row groups at every row-group boundary and drops those that can
+no longer contribute — zero IO and zero decode for the skipped row groups
+([#22450]). This release also adds multi-column lexicographic statistics
+reordering ([#23888]), so compound `ORDER BY` queries benefit too. In the
+`topk_tpch` benchmark suite, 5 of 11 queries got 3-4x faster with no
+regressions, reducing total suite runtime by 44%.
+Thanks to [@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+
+### Per-Partition TopK for Window Functions
+
+A common analytics pattern selects the top N rows per group using a window
+function:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+DataFusion previously sorted the *entire* input to evaluate the window
+function, even though only a handful of rows per partition survive the filter.
+DataFusion 55 recognizes this pattern and uses a new `PartitionedTopKExec`
+operator that keeps only the top N rows per partition, dramatically reducing
+sorting and memory for high-cardinality inputs. The optimization applies to
+`ROW_NUMBER` and `RANK`, resolving a feature request first filed in 2023
+([#6899]).
+Thanks to [@SubhamSinghal] for implementing this feature, with reviews from
+[@2010YOUY01] and [@kosiew]. Related PRs: [#21479], [#22885], [#23096]
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type dragged
+the entire grouping onto a slower row-encoded fallback. DataFusion 55 completes
+the type coverage ([#22715]): new specializations were added for `Decimal256`,
+`Float16`, and `Interval` ([#23849], [#23785], [#23786]), and a new generic
+`Rows`-backed `GroupColumn` keeps mixed schemas on the column-wise path
+([#23523]).
+Thanks to [@zhuqi-lucas] and [@tohuya6] for this work.
+
+**Split Aggregation Streams**:
+The hash aggregation state machine previously handled partial aggregation,
+final aggregation, and streaming cases in one shared implementation. DataFusion
+55 splits these semantically distinct paths into dedicated streams ([#22729]),
+making the code easier to optimize and extend (part of epic [#22710]).
+Thanks to [@2010YOUY01] for this work, with reviews from [@Rachelint] and
+[@alamb].
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up pays off
+across many workloads. This release optimizes many, including [find_in_set]
+(up to 24x faster), [trunc] (10x), [replace] (2x), [regexp_instr] (40%),
+[regexp_match], [round], [date_trunc], [date_part], [get_field], [upper], and
+[string_trim], plus dictionary-encoding preservation for many string functions
+([#23743], [#23930], [#24100]) and a 100x improvement to [approx_distinct] for
+low-cardinality inputs with many groups ([#22768]). The `approx_distinct`
+aggregate also gained support for many more types, including `Decimal`,
+`Interval`, `Duration`, `Struct`, `Map`, and `Union`, thanks to [@mkleen].
+Thanks to the many contributors who drove this work, especially [@andygrove],
+[@neilconway], [@lyne7-sc], [@theirix], and [@haohuaijin].
+
+**Faster `IN` List Evaluation**:
+`IN` list membership checks can run millions of times during a scan, especially
+with dynamic filter pushdown. DataFusion 55 adds exact lookup strategies
+selected by type and list size, including bitmap filters for small integer
+types and branchless filters for small primitive lists ([#19241]).
+Thanks to [@geoffreyclaude] for driving this work, with contributions from
+[@alamb]. Related PRs: [#23012], [#23014], [#23299]
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The `EnforceDistribution` and `EnforceSorting` physical optimizer passes are
+now merged into a single `EnsureRequirements` pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+**Smarter Join Planning**:
+DataFusion 55 converts inner joins to semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for this work.
+
+**Scalar UDF Strictness Metadata**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]). The optimizer uses this metadata to prove that
+filters reject `NULL`s, unlocking outer join elimination for queries that
+filter on the result of a function call.
+Thanks to [@lyne7-sc] for implementing this feature, with reviews from [@alamb]
+and [@kosiew].
+
+**Faster Optimizer Passes**:
+The logical optimizer now skips subquery traversal when there are no subqueries
+and rewrites plans in place ([#22298]), collapses chained projections in a
+single pass ([#22389]), and avoids re-inlining expensive common subexpressions
+during projection pushdown ([#23459]).
+Thanks to [@adriangb], [@Dandandan], and [@fordN] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+When a table declares a nested column narrower than the Parquet file's physical
+type, DataFusion previously read every leaf of the column and dropped the extra
+subfields in memory. DataFusion 55 derives the projection mask through casts,
+so only the leaves that are actually accessed are read — one production query
+reported by the [DataFusion Comet] project went from reading 1.35 TB to reading
+only the required data ([#24090]).
+Thanks to [@mbutrovich] for this work, with reviews from [@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+`ParquetMetaData` clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## New Features ✨
+
+### Range Partitioning
+
+DataFusion previously supported only hash and round-robin repartitioning.
+DataFusion 55 adds *range partitioning*, where rows are distributed to
+partitions based on ordered split points ([#22395], design discussion
+[#21992]). Range partitioning preserves ordering across partitions, which is a
+natural fit for pre-sorted data and for distributed engines that shuffle by
+range.
Review Comment:
@gene-bordegaray -- I am very behind on this post (I have been working on the ALP blog post with @sdf-jkl https://github.com/apache/parquet-site/pull/195)
I now have some time and will likely focus on this over the next day or two
I also added you as a collaborator on my fork, so you should now be able to push directly to this branch -- please feel free to do so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5413370052
> I think you should also mention that you can now opt in for better ordering equivalence (for example custom cast)
>
> * [`ScalarUdfImpl::strictly_order_preserving`: Allow expression to report whether they keep the same ordering of the input datafusion#23807](https://github.com/apache/datafusion/pull/23807)
Done in https://github.com/apache/datafusion-site/pull/203/commits/1aed4ff093c86993605e454bc896ba13870d9871 and https://github.com/apache/datafusion-site/pull/203/commits/f1dc80f61546b7b00191f9039b3b4f3088709193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5413022452
FYI your work is mentioned in this post:
@2010YOUY01, @AdamGS, @adriangb, @andygrove, @athlcode, @avantgardnerio, @blinding-pixels, @comphead, @crm26, @Dandandan, @davidlghellin, @EdsonPetry, @EmilyMatt, @fordN, @gene-bordegaray, @geoffreyclaude, @gmhelmold, @haohuaijin,
@jayshrivastava, @JeelRajodiya, @jkylling, @joroKr21, @JSOD11, @KarpagamKarthikeyan, @kosiew, @kumarUjjawal, @lyne7-sc, @mattp5657, @maxburke, @mbutrovich, @mithuncy, @mkleen, @neilconway, @niebayes, @pantShrey, @pepijnve, @peterxcli, @Rachelint,
@Rich-T-kid, @saadtajwar, @simonvandel, @sjhddh, @stuhood, @SubhamSinghal, @theirix, @timsaucer, @tohuya6, @wirybeaver, @yinli-systems, @zhuqi-lucas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#discussion_r3868106602
##########
content/blog/2026-08-25-datafusion-55.0.0.md:
##########
@@ -0,0 +1,719 @@
+---
+layout: post
+title: Apache DataFusion 55.0.0 Released
+date: 2026-08-25
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+We are proud to announce the release of [DataFusion 55.0.0]. This post
+highlights some of the many improvements since [DataFusion 54.0.0], such as
+significant performance increases, range partitioning, `MERGE INTO` support, and
+runtime row-group pruning. The complete list of changes is available in the
+[changelog]. This release represents roughly 10 weeks of development and 877
+commits. Thanks to the [175 contributors] (a new record!) for making it
+possible.
+
+[DataFusion 55.0.0]: https://crates.io/crates/datafusion/55.0.0
+[DataFusion 54.0.0]: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0/
+[changelog]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md
+[175 contributors]: https://github.com/apache/datafusion/blob/branch-55/dev/changelog/55.0.0.md#credits
+
+<img
+src="/blog/images/datafusion-55.0.0/commits_contributors.svg"
+width="100%"
+class="img-fluid"
+alt="Bar charts showing total commits, commits per day, and unique contributors for DataFusion releases 53.0.0, 54.0.0, and 55.0.0."
+/>
+
+**Figure 1**: Development activity over the last three DataFusion releases:
+total commits, commits per day, and unique contributors, computed from each
+release's [changelog] and release dates.
+
+## Performance Improvements 🚀
+
+In this release, we focused our optimizations on making DataFusion faster across
+the board rather than further optimizing our very satisfying ClickBench numbers
+(DataFusion is already the fastest in some cases — see the
+[appendix]), as ClickBench represents only a tiny fraction of what our actual users
+do (e.g. its files have no page index and contain only integer and string columns).
+
+Here is a representative sample of the performance improvements in this release;
+see the [full list in the appendix][perf appendix].
+
+| Improvement | Representative Result | Area |
+|-----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| Runtime row-group pruning for TopK | [4.2x faster `topk_tpch` Q8](https://github.com/apache/datafusion/pull/22450#issuecomment-4720594338) | Sort / TopK |
+| Faster `IN` list evaluation | [up to 14.7x faster for small primitive lists](https://github.com/apache/datafusion/pull/23014), [9.7x faster for `UInt8`](https://github.com/apache/datafusion/pull/23011) | Expressions |
+| Prune unread Parquet leaves for nested columns | [Reduces reads from 1.35 TB to 30.9 GB in a production Comet query](https://github.com/apache/datafusion/pull/24090) | Scan / IO |
+| Fewer object store requests for CSV | [70% faster TPC-H CSV with simulated latency](https://github.com/apache/datafusion/pull/22962#issuecomment-4721729807) | Scan / IO |
+| Faster `SortPreservingMerge` tie-breaker | [8% faster `sort_tpch` Q6](https://github.com/apache/datafusion/pull/23107#issuecomment-4776877963) | Sorting |
+| Native `GROUP BY` on `FixedSizeBinary` (e.g. UUIDs) | [~5% faster grouping 200M UUIDs](https://github.com/apache/datafusion/pull/23646#pullrequestreview-4900163566), [much less memory](https://github.com/apache/datafusion/pull/23646#issuecomment-4996436765) | Aggregation |
+
+
+### Sort Pushdown + TopK Pruning
+
+The multi-release [Sort Pushdown effort] continues to optimize `ORDER BY` and
+`ORDER BY ... LIMIT` (TopK) queries. In DataFusion 55, as a dynamic filter
+threshold tightens, the Parquet reader re-evaluates the threshold against the
+remaining row groups and drops those that can no longer contribute ([#22450]),
+and compound `ORDER BY` queries are now supported. Together these reduce the
+total `topk_tpch` suite runtime by ~43%; see our
+[Optimizing for Almost Sorted Data] blog post for more details. Thanks to
+[@zhuqi-lucas] for driving this work, with reviews from [@adriangb].
+
+[Sort Pushdown effort]: https://github.com/apache/datafusion/issues/23036
+[Optimizing for Almost Sorted Data]: https://datafusion.apache.org/blog/2026/07/20/sort-pushdown/
+
+### Aggregation Improvements
+
+**Complete Multi-Column `GROUP BY` Type Coverage**:
+DataFusion's column-wise `GROUP BY` storage (`GroupValuesColumn`) has
+type-specific fast paths, but previously any unsupported column type forced
+the entire grouping onto a slower row-encoded fallback. For example, this
+query to deduplicate a table of UUIDs used to hit the slow path:
+
+```sql
+SELECT count(*) FROM (SELECT uuid, id FROM 'uuids.parquet' GROUP BY uuid, id);
+```
+
+DataFusion 55 completes the type coverage ([#22715]), so the query above now runs about 5%
+faster on 200M UUIDs, and uses much less memory (see [#23645]). Thanks to
+[@zhuqi-lucas], [@tohuya6], and [@maxburke] for this work.
+
+### Faster Functions
+
+DataFusion ships hundreds of built-in functions, so speeding them up improves performance
+for many workloads. This release optimizes dozens of functions — up to 24x faster
+for [`find_in_set`][find_in_set] and 100x for [`approx_distinct`][approx_distinct]
+with low-cardinality inputs and many groups ([#22768]). It also includes
+dictionary-encoding preservation for many string functions ([#23743],
+[#23930], [#24100]) and new `IN` list specializations, such as bitmap filters for small integer types ([#19241]). See the
+[full list in the appendix][perf appendix].
+Thanks to the many contributors who drove this work, especially
+[@andygrove], [@geoffreyclaude], [@neilconway], [@lyne7-sc], [@theirix], and
+[@haohuaijin].
+
+[perf appendix]: #appendix-full-list-of-performance-improvements
+
+### Planner Improvements
+
+**Unified Distribution and Sorting Enforcement**:
+The [`EnforceDistribution`][EnforceDistribution] and
+[`EnforceSorting`][EnforceSorting] physical optimizer passes are
+now merged into a single [`EnsureRequirements`][EnsureRequirements] pass with idempotent sort
+pushdown ([#21976]), fixing longstanding ordering issues between the two passes
+and enabling the sort pushdown work described above.
+Thanks to [@zhuqi-lucas] for this work, with reviews from [@2010YOUY01] and
+[@alamb].
+
+[EnforceDistribution]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_distribution/struct.EnforceDistribution.html
+[EnforceSorting]: https://docs.rs/datafusion/54.0.0/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html
+[EnsureRequirements]: https://docs.rs/datafusion/55.0.0/datafusion/physical_optimizer/ensure_requirements/struct.EnsureRequirements.html
+
+**Smarter Join Planning**:
+DataFusion 55 now converts inner joins to more efficient semi joins when equivalent ([#22652]),
+eliminates `LEFT`/`RIGHT` joins with redundant sides ([#23566]), handles
+intermediate projections in outer join elimination ([#22534]), and reorders
+predicates in conjunctions using a cost heuristic ([#22343]).
+Thanks to [@neilconway] and [@simonvandel] for driving this work.
+
+**Better Scalar UDF Metadata APIs**:
+Scalar UDFs can now declare that they are *strict* (they return `NULL` when any
+input is `NULL`) ([#23148]), letting the optimizer eliminate outer joins for
+queries that filter on a function result, and *strictly order preserving*
+(sorted input yields identically sorted output) ([#23807]), letting the
+optimizer eliminate redundant sorts on expressions such as custom casts.
+Thanks to [@lyne7-sc] and [@rluvaton] for this work, with reviews from
+[@alamb], [@kosiew], and [@getChan].
+
+**Faster Optimizer**:
+The optimizer continues to get faster, with improvements such as selective
+subquery traversal and in-place rewrites ([#22298]), collapsing chained
+projections ([#22389]), avoiding re-inlining expensive common subexpressions
+([#23459]), and a faster `PushDownFilter` rule that modifies plans in place
+rather than copying them ([#20002], [#21668]).
+Thanks to [@adriangb], [@Dandandan], [@fordN], and [@joroKr21] for this work.
+
+### Scan Improvements
+
+**Pruning Unread Parquet Leaves for Nested Columns**:
+
+Systems that embed DataFusion — such as [DataFusion Comet], [delta-rs], and
+Iceberg integrations — often hand DataFusion a table schema that includes only the nested
+subfields the query needs. For example, given a file whose `events` column
+physically holds four subfields, a table might declare only two of them:
+
+```sql
+-- events column is ARRAY<STRUCT<id BIGINT, name VARCHAR, payload VARCHAR, trace VARCHAR>>
+-- Table definition only refers to the first two subfields, id and name
+CREATE EXTERNAL TABLE events (
+ events ARRAY<STRUCT<id BIGINT, name VARCHAR>>
+)
+STORED AS PARQUET LOCATION 'events.parquet';
+```
+
+DataFusion correctly reconciles these schemas, but prior to DataFusion 55, all
+four leaves were read from the file and decoded, including the large `payload`
+and `trace` subfields, which were then thrown away. The Comet project reported
+a production query where this extra decoding caused 1.35 TB of reads, whereas
+plain Spark read only 30.9 GB for the same pruned schema. DataFusion 55 closes
+that gap by not reading the undeclared `payload` and `trace` leaves from the
+file at all ([#24090]). Thanks to [@mbutrovich] for this work, with reviews from
+[@adriangb].
+
+[DataFusion Comet]: https://datafusion.apache.org/comet/
+[delta-rs]: https://github.com/delta-io/delta-rs
+
+**Other Scan Improvements**:
+DataFusion 55 also skips loading the page index (and an expensive
+[`ParquetMetaData`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html) clone) when a file has no page index ([#24150]), supports
+file-level Parquet row selections ([#22940]), and lowers the default
+`repartition_file_min_size` from 10 MiB to 1 MiB for better parallelism on
+small files ([#22439]).
+Thanks to [@alamb], [@haohuaijin], and [@adriangb].
+
+## Stability Improvements 🛡️
+
+The community also improved DataFusion's handling of larger-than-memory
+aggregate workloads (e.g. [#23657], [#23965], [#24061]), building on a
+refactoring of the aggregation path into dedicated streams (epic [#22710]).
+Sorts under memory pressure are more resilient: when a spill
+merge cannot reserve enough memory, DataFusion now re-spills the largest stream
+in smaller batches rather than failing ([#22945]), and caps the merge fan-in to
+bound memory use ([#23066]). Thanks to [@2010YOUY01], [@EmilyMatt],
+[@yinli-systems], [@Rachelint], and [@pepijnve] (who fixed a subtle lost-wakeup
+bug in the spill pool, [#23522]) for this work.
+
+## New Features ✨
+
+### `file_row_index()` and `input_file_name()`
+
+DataFusion 55 adds [`file_row_index`][file_row_index] ([#22604]) and [`input_file_name`][input_file_name] ([#22978]) functions
+to expose Parquet virtual columns:
+
+```sql
+> select *, input_file_name(), file_row_index() from '/tmp/foo.parquet';
++---------+-------------------+------------------+
+| column1 | input_file_name() | file_row_index() |
++---------+-------------------+------------------+
+| 100 | tmp/foo.parquet | 0 |
+| 200 | tmp/foo.parquet | 1 |
++---------+-------------------+------------------+
+```
+
+Such functions are useful for change data capture, debugging, and
+Spark-compatible workloads. Thanks to [@mbutrovich] and [@AdamGS] for this work
+(reviving earlier work from [@jkylling]), with reviews from [@adriangb],
+[@comphead], and [@niebayes].
+
+### Range Partitioning
+
+DataFusion 55 adds native *range partitioning* support, which maps rows to partitions by key ranges (rather than hash
+values). Query inputs are often range partitioned in real-world scenarios, such as time-series data
+written as one file per day or hour. DataFusion uses range partitioning information to
+avoid expensive repartitioning operations and push more specific dynamic filters
+to scans.
+
+Data that is range partitioned declares an ordering and a list of split
+points. Partition `i` holds the keys that fall between split point `i-1` and
+split point `i`:
+
+```text
+ordering = [date ASC NULLS LAST]
+split_points = [(2022-01-01), (2023-01-01)]
+
+partition 0: date < 2022-01-01
+partition 1: 2022-01-01 <= date < 2023-01-01
+partition 2: date >= 2023-01-01
+```
+
+For more details, please see the documentation for
+[`Partitioning::Range`][Partitioning::Range], the planning epic ([#22395]), and
+the design discussion ([#21992]). Thanks to [@gene-bordegaray], [@saadtajwar], [@peterxcli], [@stuhood],
+[@gmhelmold], [@mattp5657], [@mithuncy], [@JSOD11], [@EdsonPetry],
+[@Rich-T-kid], and [@blinding-pixels] for driving this substantial community
+effort.
+
+### `MERGE INTO` Planner Support
+
+`MERGE INTO` (SQL:2003) is a widely used DML statement for upsert and
+conditional update workloads, and a key building block for table formats such
+as Apache Iceberg and Delta Lake. DataFusion 55 adds the logical plan types
+([#20763]) along with SQL planner and physical planner support, and a new
+[`TableProvider::merge_into`][TableProvider::merge_into] hook ([#22988]) so table implementations can
+execute merge operations:
+
+```sql
+MERGE INTO target t
+USING source s
+ON t.id = s.id
+WHEN MATCHED AND s.deleted THEN DELETE
+WHEN MATCHED THEN UPDATE SET name = s.name
+WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
+```
+
+Built-in table providers do not yet implement the hook, but custom
+[`TableProvider`][TableProvider] implementations (such as lakehouse table formats) can now plug
+in their own merge execution.
Review Comment:
Thanks for highlighting MERGE INTO. One small note: 55.0 still rejects two target-alias edge cases—target-correlated subqueries and a source qualifier matching the target’s real table name. apache/datafusion#24429 fixes both and is planned for 55.1 under apache/datafusion#24462. Would it be useful to mention this briefly as a known limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5413140494
This is great, @alamb! Nice work. And congrats to everyone on another awesome release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5451049923
Publishing workflow seems to be waiting on a builder: https://github.com/apache/datafusion-site/actions/runs/33159387574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add DataFusion 55.0.0 release blog post [datafusion-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #203:
URL: https://github.com/apache/datafusion-site/pull/203#issuecomment-5451652838
Post is live: https://datafusion.apache.org/blog/output/2026/08/25/datafusion-55.0.0/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org