You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/29 16:11:16 UTC
(datafusion) branch gh-readonly-queue/main/pr-21824-66980e26df75740006229125acbf6815ee04a9f4 created (now f0c53066f0)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-21824-66980e26df75740006229125acbf6815ee04a9f4
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at f0c53066f0 docs(optimizer): add generated optimizer rules reference (#21824)
This branch includes the following new commits:
new f0c53066f0 docs(optimizer): add generated optimizer rules reference (#21824)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: docs(optimizer): add generated optimizer rules reference (#21824)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-21824-66980e26df75740006229125acbf6815ee04a9f4
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit f0c53066f0f4977167b25c98296b9b4183f96524
Author: Kumar Ujjawal <uj...@gmail.com>
AuthorDate: Wed Apr 29 21:40:52 2026 +0530
docs(optimizer): add generated optimizer rules reference (#21824)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #21771.
## Rationale for this change
There is no reference page that lists the built-in analyzer, logical
optimizer, and physical optimizer rules. This PR adds that missing
reference.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
- Add a generated optimizer rules reference page.
- Add a renderer and generator binary for the optimizer rules docs.
- Keep the rule documentation metadata private in core so this does not
add new public optimizer APIs.
- Link the new page from the query optimizer guide and the docs index.
- Add tests that check the documented rule order matches the default
analyzer, logical optimizer, and physical optimizer pipelines.
- Fix the `eliminate_join` description so it matches the actual rule
behavior.
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
Yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
No Public API Change
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Oleks V <co...@users.noreply.github.com>
---
datafusion/core/Cargo.toml | 2 +-
datafusion/core/src/lib.rs | 4 +
datafusion/core/src/optimizer_rule_reference.md | 93 +++++++++++++++++++++++
datafusion/core/src/optimizer_rule_reference.rs | 85 +++++++++++++++++++++
docs/source/library-user-guide/query-optimizer.md | 4 +
5 files changed, 187 insertions(+), 1 deletion(-)
diff --git a/datafusion/core/Cargo.toml b/datafusion/core/Cargo.toml
index 0a3783de9d..ebd05392c9 100644
--- a/datafusion/core/Cargo.toml
+++ b/datafusion/core/Cargo.toml
@@ -19,7 +19,7 @@
name = "datafusion"
description = "DataFusion is an in-memory query engine that uses Apache Arrow as the memory model"
keywords = ["arrow", "query", "sql"]
-include = ["benches/*.rs", "src/**/*.rs", "Cargo.toml", "LICENSE.txt", "NOTICE.txt"]
+include = ["benches/*.rs", "src/**/*.md", "src/**/*.rs", "Cargo.toml", "LICENSE.txt", "NOTICE.txt"]
readme = "../../README.md"
version = { workspace = true }
edition = { workspace = true }
diff --git a/datafusion/core/src/lib.rs b/datafusion/core/src/lib.rs
index 1d8368f54b..3170f4be7f 100644
--- a/datafusion/core/src/lib.rs
+++ b/datafusion/core/src/lib.rs
@@ -761,6 +761,7 @@
//! [`RecordBatch`]: arrow::array::RecordBatch
//! [`RecordBatchReader`]: arrow::record_batch::RecordBatchReader
//! [`Array`]: arrow::array::Array
+#![doc = include_str!("optimizer_rule_reference.md")]
extern crate core;
#[cfg(feature = "sql")]
@@ -786,6 +787,9 @@ pub use parquet;
#[cfg(feature = "avro")]
pub use datafusion_datasource_avro::arrow_avro;
+#[cfg(test)]
+mod optimizer_rule_reference;
+
// re-export DataFusion sub-crates at the top level. Use `pub use *`
// so that the contents of the subcrates appears in rustdocs
// for details, see https://github.com/apache/datafusion/issues/6648
diff --git a/datafusion/core/src/optimizer_rule_reference.md b/datafusion/core/src/optimizer_rule_reference.md
new file mode 100644
index 0000000000..fcbb200c71
--- /dev/null
+++ b/datafusion/core/src/optimizer_rule_reference.md
@@ -0,0 +1,93 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+## Built-in Optimizer Rules
+
+DataFusion applies a default analyzer, logical optimizer, and physical
+optimizer pipeline.
+
+The rule names listed here match the names shown by `EXPLAIN VERBOSE`.
+
+Rule order matters. The default pipeline may change between releases.
+
+### Analyzer Rules
+
+| order | rule | summary |
+| ----- | --------------------------- | --------------------------------------------------------------------------------------- |
+| 1 | `resolve_grouping_function` | Rewrites `GROUPING(...)` calls into expressions over DataFusion's internal grouping id. |
+| 2 | `type_coercion` | Adds implicit casts so operators and functions receive valid input types. |
+
+### Logical Optimizer Rules
+
+| order | rule | summary |
+| ----- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| 1 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |
+| 2 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |
+| 3 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |
+| 4 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |
+| 5 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |
+| 6 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |
+| 7 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |
+| 8 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |
+| 9 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |
+| 10 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |
+| 11 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |
+| 12 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |
+| 13 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |
+| 14 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |
+| 15 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |
+| 16 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |
+| 17 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits. |
+| 18 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
+| 19 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
+| 20 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
+| 21 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
+| 22 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
+| 23 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
+| 24 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
+
+### Physical Optimizer Rules
+
+The same rule name may appear more than once when the default pipeline runs it
+in multiple phases.
+
+| order | rule | phase | summary |
+| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
+| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |
+| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |
+| 3 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
+| 4 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
+| 5 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
+| 6 | `EnforceDistribution` | - | Adds repartitioning only where needed to satisfy physical distribution requirements. |
+| 7 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |
+| 8 | `EnforceSorting` | - | Adds or removes local sorts to satisfy required input orderings. |
+| 9 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |
+| 10 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |
+| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
+| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
+| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
+| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |
+| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |
+| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |
+| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |
+| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |
+| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |
+| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |
+| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |
+| 22 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
diff --git a/datafusion/core/src/optimizer_rule_reference.rs b/datafusion/core/src/optimizer_rule_reference.rs
new file mode 100644
index 0000000000..64db51b290
--- /dev/null
+++ b/datafusion/core/src/optimizer_rule_reference.rs
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_optimizer::analyzer::Analyzer;
+use datafusion_optimizer::optimizer::Optimizer;
+use datafusion_physical_optimizer::optimizer::PhysicalOptimizer;
+
+const OPTIMIZER_RULE_REFERENCE: &str = include_str!("optimizer_rule_reference.md");
+
+fn documented_rules(section_heading: &str) -> Vec<String> {
+ let mut in_section = false;
+ let mut names = vec![];
+
+ for line in OPTIMIZER_RULE_REFERENCE.lines() {
+ if line == section_heading {
+ in_section = true;
+ continue;
+ }
+
+ if in_section && line.starts_with("### ") {
+ break;
+ }
+
+ if !in_section || !line.starts_with('|') || line.contains("---") {
+ continue;
+ }
+
+ let columns: Vec<_> = line.split('|').map(str::trim).collect();
+
+ if columns.len() < 4 || columns[1] == "order" {
+ continue;
+ }
+
+ names.push(columns[2].trim_matches('`').to_string());
+ }
+
+ names
+}
+
+#[test]
+fn analyzer_rules_match_documented_order() {
+ let rules: Vec<_> = Analyzer::new()
+ .rules
+ .iter()
+ .map(|rule| rule.name().to_string())
+ .collect();
+
+ assert_eq!(documented_rules("### Analyzer Rules"), rules);
+}
+
+#[test]
+fn logical_rules_match_documented_order() {
+ let rules: Vec<_> = Optimizer::new()
+ .rules
+ .iter()
+ .map(|rule| rule.name().to_string())
+ .collect();
+
+ assert_eq!(documented_rules("### Logical Optimizer Rules"), rules);
+}
+
+#[test]
+fn physical_rules_match_documented_order() {
+ let rules: Vec<_> = PhysicalOptimizer::new()
+ .rules
+ .iter()
+ .map(|rule| rule.name().to_string())
+ .collect();
+
+ assert_eq!(documented_rules("### Physical Optimizer Rules"), rules);
+}
diff --git a/docs/source/library-user-guide/query-optimizer.md b/docs/source/library-user-guide/query-optimizer.md
index 2254776bf6..a3853d39dd 100644
--- a/docs/source/library-user-guide/query-optimizer.md
+++ b/docs/source/library-user-guide/query-optimizer.md
@@ -28,6 +28,9 @@ This crate is a submodule of DataFusion that provides a query optimizer for logi
contains an extensive set of [`OptimizerRule`]s and [`PhysicalOptimizerRule`]s that may rewrite the plan and/or its expressions so
they execute more quickly while still computing the same result.
+For a reference list of the built-in analyzer, logical optimizer, and physical optimizer rules,
+see [Optimizer Rule Reference].
+
For a deeper background on optimizer architecture and rule types and predicates, see
[Optimizing SQL (and DataFrames) in DataFusion, Part 1], [Part 2],
[Using Ordering for Better Plans in Apache DataFusion], and
@@ -39,6 +42,7 @@ For a deeper background on optimizer architecture and rule types and predicates,
[part 2]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two
[using ordering for better plans in apache datafusion]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis
[dynamic filters: passing information between operators during execution for 25x faster queries]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters
+[optimizer rule reference]: https://docs.rs/datafusion/latest/datafusion/index.html#built-in-optimizer-rules
[`logicalplan`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html
## Running the Optimizer
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org