You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/02/02 19:06:42 UTC
[PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
adriangb opened a new pull request, #20117:
URL: https://github.com/apache/datafusion/pull/20117
## Summary
This PR adds a new optimizer rule `ExtractLeafExpressions` that extracts `MoveTowardsLeafNodes` sub-expressions (like `get_field`) from Filter, Sort, Limit, Aggregate, and **Projection** nodes into intermediate projections.
This normalization allows `OptimizeProjections` (which runs next) to merge consecutive projections and push `get_field` expressions down to the scan, enabling Parquet column pruning for struct fields.
### Example
```sql
SELECT id, s['label'] FROM t WHERE s['value'] > 150
```
**Before:** `get_field(s, 'label')` stayed in ProjectionExec, reading full struct `s`
**After:** Both `get_field` expressions pushed to DataSourceExec:
```
DataSourceExec: projection=[get_field(s, value) as __leaf_5, get_field(s, label) as __leaf_4, id]
```
### How It Works
The rule:
1. Extracts `MoveTowardsLeafNodes` expressions into `__leaf_N` aliases
2. Creates inner projections with extracted expressions + pass-through columns
3. Creates outer projections to restore original schema names
4. Handles deduplication of identical expressions
5. Skips expressions already aliased with `__leaf_*` to ensure idempotency
### Interaction with OptimizeProjections
**Optimizer order** (from `optimizer.rs`):
```rust
Arc::new(ExtractLeafExpressions::new()), // Runs FIRST
Arc::new(OptimizeProjections::new()), // Runs SECOND - merges projections
```
## Test plan
- [x] New unit tests for projection extraction in `extract_leaf_expressions.rs`
- [x] Updated sqllogictest expectations in `projection_pushdown.slt`
- [x] All optimizer tests pass (`cargo test -p datafusion-optimizer`)
🤖 Generated with [Claude Code](https://claude.ai/code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2760254821
##########
datafusion/sqllogictest/test_files/push_down_filter.slt:
##########
@@ -116,11 +116,12 @@ explain select * from (select column1, unnest(column2) as o from d) where o['a']
----
physical_plan
01)ProjectionExec: expr=[column1@0 as column1, __unnest_placeholder(d.column2,depth=1)@1 as o]
-02)--FilterExec: get_field(__unnest_placeholder(d.column2,depth=1)@1, a) = 1
+02)--FilterExec: __datafusion_extracted_1@0 = 1, projection=[column1@1, __unnest_placeholder(d.column2,depth=1)@2]
03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-04)------UnnestExec
-05)--------ProjectionExec: expr=[column1@0 as column1, column2@1 as __unnest_placeholder(d.column2)]
-06)----------DataSourceExec: partitions=1, partition_sizes=[1]
+04)------ProjectionExec: expr=[get_field(__unnest_placeholder(d.column2,depth=1)@1, a) as __datafusion_extracted_1, column1@0 as column1, __unnest_placeholder(d.column2,depth=1)@1 as __unnest_placeholder(d.column2,depth=1)]
+05)--------UnnestExec
Review Comment:
I think it would be quite complex to try to push the get_filter through the unnest, and ultimately I don't think Parquet would be able to optimize the scan (maybe I'm wrong about this?) so there would be little point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843241718
BTW codex found a test that shows a single projection being extracted doesn't get pushed down
I can make this a separate PR if you like
```sql
###
# Test 2.1b: Projection-only get_field (potential optimization target)
###
query TT
EXPLAIN SELECT s['label'] FROM simple_struct;
----
logical_plan
01)Projection: get_field(simple_struct.s, Utf8("label"))
02)--TableScan: simple_struct projection=[s]
physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, label) as simple_struct.s[label]], file_type=parquet
# Verify correctness
query T
SELECT s['label'] FROM simple_struct ORDER BY s['label'];
----
alpha
beta
delta
epsilon
gamma
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843684448
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2801263356
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -95,9 +101,533 @@ impl OptimizerRule for ExtractLeafExpressions {
fn rewrite(
&self,
plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
+ config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
- Ok(Transformed::no(plan))
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// Works for any number of inputs (0, 1, 2, …N). For multi-input nodes
+/// like Join, each extracted sub-expression is routed to the correct input
+/// by checking which input's schema contains all of the expression's column
+/// references.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Only extract from plan types whose output schema is predictable after
+ // expression rewriting. Nodes like Window derive column names from
+ // their expressions, so rewriting `get_field` inside a window function
+ // changes the output schema and breaks the recovery projection.
+ if !matches!(
+ &plan,
+ LogicalPlan::Aggregate(_)
+ | LogicalPlan::Filter(_)
+ | LogicalPlan::Sort(_)
+ | LogicalPlan::Limit(_)
+ | LogicalPlan::Join(_)
+ ) {
+ return Ok(Transformed::no(plan));
+ }
+
+ let inputs = plan.inputs();
+ if inputs.is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ // Save original output schema before any transformation
+ let original_schema = Arc::clone(plan.schema());
+
+ // Build per-input schemas from borrowed inputs (before plan is consumed
+ // by map_expressions). We only need schemas and column sets for routing;
+ // the actual inputs are cloned later only if extraction succeeds.
+ let input_schemas: Vec<Arc<DFSchema>> =
+ inputs.iter().map(|i| Arc::clone(i.schema())).collect();
+
+ // Build per-input extractors
+ let mut extractors: Vec<LeafExpressionExtractor> = input_schemas
+ .iter()
+ .map(|schema| LeafExpressionExtractor::new(schema.as_ref(), alias_generator))
+ .collect();
+
+ // Build per-input column sets for routing expressions to the correct input
+ let input_column_sets: Vec<std::collections::HashSet<Column>> = input_schemas
+ .iter()
+ .map(|schema| schema_columns(schema.as_ref()))
+ .collect();
+
+ // Transform expressions via map_expressions with routing
+ let transformed = plan.map_expressions(|expr| {
+ routing_extract(expr, &mut extractors, &input_column_sets)
+ })?;
+
+ // If no expressions were rewritten, nothing was extracted
+ if !transformed.transformed {
+ return Ok(transformed);
+ }
+
+ // Clone inputs now that we know extraction succeeded. Wrap in Arc
+ // upfront since build_extraction_projection expects &Arc<LogicalPlan>.
+ let owned_inputs: Vec<Arc<LogicalPlan>> = transformed
+ .data
+ .inputs()
+ .into_iter()
+ .map(|i| Arc::new(i.clone()))
+ .collect();
+
+ // Build per-input extraction projections (None means no extractions for that input)
+ let new_inputs: Vec<LogicalPlan> = owned_inputs
+ .into_iter()
+ .zip(extractors.iter())
+ .map(|(input_arc, extractor)| {
+ match extractor.build_extraction_projection(&input_arc)? {
+ Some(plan) => Ok(plan),
+ // No extractions for this input — recover the LogicalPlan
+ // without cloning (refcount is 1 since build returned None).
+ None => {
+ Ok(Arc::try_unwrap(input_arc).unwrap_or_else(|arc| (*arc).clone()))
+ }
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // Rebuild the plan keeping its rewritten expressions but replacing
+ // inputs with the new extraction projections.
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Add recovery projection if the output schema changed
+ let recovered = build_recovery_projection(original_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Given an expression, returns the index of the input whose columns fully
+/// cover the expression's column references.
+/// Returns `None` if the expression references columns from multiple inputs
+/// or if multiple inputs match (ambiguous, e.g. unqualified columns present
+/// in both sides of a join).
+fn find_owning_input(
+ expr: &Expr,
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Option<usize> {
+ let mut found = None;
+ for (idx, cols) in input_column_sets.iter().enumerate() {
+ if has_all_column_refs(expr, cols) {
+ if found.is_some() {
+ // Ambiguous — multiple inputs match
+ return None;
+ }
+ found = Some(idx);
+ }
+ }
+ found
+}
+
+/// Walks an expression tree top-down, extracting `MoveTowardsLeafNodes`
+/// sub-expressions and routing each to the correct per-input extractor.
+fn routing_extract(
+ expr: Expr,
+ extractors: &mut [LeafExpressionExtractor],
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Result<Transformed<Expr>> {
+ expr.transform_down(|e| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &e
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: e,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
Review Comment:
I think it'd be just as likely that users create some metadata marker as that they create aliases like `__datafusion_extracted`. There is prior art for using "special" aliases like this in the CSE rule. I don't love it but it does work. I could add an extra check for metadata but let's leave that for the day a user reports an error with the name detection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2760375603
##########
datafusion/expr/src/expr.rs:
##########
Review Comment:
I really like the name `ExpressionPlacement` 👍
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -320,7 +323,7 @@ EXPLAIN SELECT id, s['value'] + 1 FROM simple_struct ORDER BY id;
----
logical_plan
01)Sort: simple_struct.id ASC NULLS LAST
-02)--Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1)
+02)--Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1) AS simple_struct.s[value] + Int64(1)
Review Comment:
I wonder if there is some way to avoid adding an alias like this when project name doesn't change. It would make the plans simpler to read.
Perhaps we can address this as a follow on PR
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1295,6 +1297,18 @@ fn rewrite_projection(
predicates: Vec<Expr>,
mut projection: Projection,
) -> Result<(Transformed<LogicalPlan>, Option<Expr>)> {
+ // Note: This check coordinates with ExtractLeafExpressions optimizer rule.
+ // See extract_leaf_expressions.rs for details on why these projections exist.
+ // Don't push filters through extracted expression projections.
+ // Pushing filters through would rewrite expressions like `__datafusion_extracted_1 > 150`
+ // back to `get_field(s,'value') > 150`, undoing the extraction.
+ if is_extracted_expr_projection(&projection) {
Review Comment:
I think the comment would be better / easier to understand the need for the special case if you included the great example from your comment
>
> ```
> Projection: col('col')
> Filter: __datafusion_extracted_1 > 1
> Projection: get_field(col, 'foo') as __datafusion_extracted_1, col
> TableScan: projection=[col]
> ```
>
> Then this rule runs and will produce:
>
> ```
> Projection: col('col')
> Projection: get_field(col, 'foo') as __datafusion_extracted_1, col
> Filter: get_field(col, 'foo') > 1
> TableScan projection=[col]
> ```
##########
datafusion/sqllogictest/test_files/explain.slt:
##########
Review Comment:
😓 that is a lot of rewrites (not related to this PR, I am just thinking about planning speed in general)
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -241,13 +242,14 @@ query TT
EXPLAIN SELECT id, s['value'] + 1 FROM simple_struct WHERE id > 2;
----
logical_plan
-01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1)
+01)Projection: simple_struct.id, __datafusion_extracted_1 + Int64(1) AS simple_struct.s[value] + Int64(1)
Review Comment:
thank you for putting these tests in first -- it is much easier to see what this PR is changing by looking at the different plans
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
Review Comment:
What does "live in projection nodes" mean here? Like that all `MoveTowardsLeafNodes` computations appear as top level Exprs in a ProjectionExec?
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -1339,7 +1359,247 @@ SELECT id, s['value'] FROM simple_struct ORDER BY id, s['value'];
5 250
#####################
-# Section 12: Cleanup
+# Section 12: Join Tests - get_field Extraction from Join Nodes
+#####################
+
+# Create a second table for join tests
+statement ok
+COPY (
+ SELECT
+ column1 as id,
+ column2 as s
+ FROM VALUES
+ (1, {role: 'admin', level: 10}),
+ (2, {role: 'user', level: 5}),
+ (3, {role: 'guest', level: 1}),
+ (4, {role: 'admin', level: 8}),
+ (5, {role: 'user', level: 3})
+) TO 'test_files/scratch/projection_pushdown/join_right.parquet'
+STORED AS PARQUET;
+
+statement ok
+CREATE EXTERNAL TABLE join_right STORED AS PARQUET
+LOCATION 'test_files/scratch/projection_pushdown/join_right.parquet';
+
+###
+# Test 12.1: Join with get_field in equijoin condition
+# Tests extraction from join ON clause - get_field on each side routed appropriately
+###
+
+query TT
+EXPLAIN SELECT simple_struct.id, join_right.id
+FROM simple_struct
+INNER JOIN join_right ON simple_struct.s['value'] = join_right.s['level'] * 10;
+----
+logical_plan
+01)Projection: simple_struct.id, join_right.id
+02)--Inner Join: __datafusion_extracted_1 = __datafusion_extracted_2 * Int64(10)
+03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
+04)------TableScan: simple_struct projection=[id, s]
+05)----Projection: get_field(join_right.s, Utf8("level")) AS __datafusion_extracted_2, join_right.id
+06)------TableScan: join_right projection=[id, s]
+physical_plan
+01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(__datafusion_extracted_1@0, __datafusion_extracted_2 * Int64(10)@2)], projection=[id@1, id@3]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet
Review Comment:
it is pretty cool to see the expression extraction pushed down on both sides of the join
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -217,13 +217,14 @@ query TT
EXPLAIN SELECT id, s['value'] FROM simple_struct WHERE id > 2;
----
logical_plan
-01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value"))
+01)Projection: simple_struct.id, __datafusion_extracted_1 AS simple_struct.s[value]
02)--Filter: simple_struct.id > Int64(2)
-03)----TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
+03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
+04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
physical_plan
-01)ProjectionExec: expr=[id@0 as id, get_field(s@1, value) as simple_struct.s[value]]
-02)--FilterExec: id@0 > 2
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, s], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+01)ProjectionExec: expr=[id@1 as id, __datafusion_extracted_1@0 as simple_struct.s[value]]
+02)--FilterExec: id@1 > 2
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_1, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
Review Comment:
Here is the optimizer pass in action -- the get_field was pushed down -- the plan looks good to me
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
Review Comment:
would be nice to make this a link too so it is checked automatically by rustdoc rather than can get out of sync
```suggestion
/// and will not push filters through them. See [`is_extracted_expr_projection`]
```
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
Review Comment:
Is there a reason to split up the comments into module on the struct?
It might make sense to leave the module level comments relatively minimal and move `#Algorithm` and everything else down to the doc comment on `ExtractLeafExpressions` so the algorithm and examples are close together
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
Review Comment:
Thank you for the comments
I think it would help to make `MoveTowardsLeafNodes` a doc link -- aka `ExpressionPlacement::MoveTowardsLeafNodes`
##########
datafusion/sqllogictest/test_files/push_down_filter.slt:
##########
@@ -116,11 +116,12 @@ explain select * from (select column1, unnest(column2) as o from d) where o['a']
----
physical_plan
01)ProjectionExec: expr=[column1@0 as column1, __unnest_placeholder(d.column2,depth=1)@1 as o]
-02)--FilterExec: get_field(__unnest_placeholder(d.column2,depth=1)@1, a) = 1
+02)--FilterExec: __datafusion_extracted_1@0 = 1, projection=[column1@1, __unnest_placeholder(d.column2,depth=1)@2]
03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-04)------UnnestExec
-05)--------ProjectionExec: expr=[column1@0 as column1, column2@1 as __unnest_placeholder(d.column2)]
-06)----------DataSourceExec: partitions=1, partition_sizes=[1]
+04)------ProjectionExec: expr=[get_field(__unnest_placeholder(d.column2,depth=1)@1, a) as __datafusion_extracted_1, column1@0 as column1, __unnest_placeholder(d.column2,depth=1)@1 as __unnest_placeholder(d.column2,depth=1)]
+05)--------UnnestExec
Review Comment:
Since it wasn't pushed through before I think it is fine that it (still) isn't pushed through
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
Review Comment:
If we used the map_expressions API, as suggested above, we would get support for Extension nodes "for free"
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::TopDown)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With TopDown traversal, we process parent nodes first, allowing us to
+/// merge expressions through child projections.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing extracted projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ let rebuilt_input = extractor.build_extraction_projection(&target, path)?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
Review Comment:
the code above seems to assume there is a single input -- so it seems strange to have code here for multiple inputs 🤔
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::TopDown)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With TopDown traversal, we process parent nodes first, allowing us to
+/// merge expressions through child projections.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
Review Comment:
I don't understand why there needs to be specialized code for different LogicalPlan types -- this seems like it is exactly the use case [`LogicalPlan::map_expressions()`](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html#method.map_expressions) is designed to handle.
Couldn't you use map_expressions to rewrite any get_field expressions, and then add the relevant projection below it?
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1295,6 +1297,18 @@ fn rewrite_projection(
predicates: Vec<Expr>,
mut projection: Projection,
) -> Result<(Transformed<LogicalPlan>, Option<Expr>)> {
+ // Note: This check coordinates with ExtractLeafExpressions optimizer rule.
+ // See extract_leaf_expressions.rs for details on why these projections exist.
+ // Don't push filters through extracted expression projections.
+ // Pushing filters through would rewrite expressions like `__datafusion_extracted_1 > 150`
+ // back to `get_field(s,'value') > 150`, undoing the extraction.
+ if is_extracted_expr_projection(&projection) {
Review Comment:
> I'd argue as a general rule there's no point in pushing a filter under a projection that is purely column selections / get_field expressions especially if we can't then push it further.
Yes I agree with this statement.
I don't really have a better suggestion other than to perhaps make the exception more general "don't push filters under projections that doesn't do computation / etc"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843001119
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing get-field-pushdown-try-3 (e95acd3972b0ef198c461aa706c091830f9b9bc3) to 9962911ee5d5cdde5f76a4489ee7d432365dde0c [diff](https://github.com/apache/datafusion/compare/9962911ee5d5cdde5f76a4489ee7d432365dde0c..e95acd3972b0ef198c461aa706c091830f9b9bc3)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=get-field-pushdown-try-3
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756630900
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -168,7 +168,7 @@ query TT
EXPLAIN SELECT id, s['value'] + 1 FROM simple_struct;
----
logical_plan
-01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1)
+01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1) AS simple_struct.s[value] + Int64(1)
Review Comment:
Note that this is not a change in the output schema name: it is already `simple_struct.s[value]`. So this is a useless alias (we can see if we can remove it) but not an actual user facing change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843386076
> 🤖: Benchmark completed
>
> Details
> ```
> group get-field-pushdown-try-3 main
> ----- ------------------------ ----
> logical_aggregate_with_join 1.01 643.3±10.62µs ? ?/sec 1.00 639.4±6.12µs ? ?/sec
> logical_select_all_from_1000 1.00 10.3±0.09ms ? ?/sec 1.12 11.5±0.15ms ? ?/sec
> logical_select_one_from_700 1.01 421.7±3.89µs ? ?/sec 1.00 415.8±2.36µs ? ?/sec
> logical_trivial_join_high_numbered_columns 1.01 380.8±12.39µs ? ?/sec 1.00 376.5±3.19µs ? ?/sec
> logical_trivial_join_low_numbered_columns 1.01 365.3±4.47µs ? ?/sec 1.00 361.5±11.63µs ? ?/sec
> physical_intersection 1.00 1627.7±31.01µs ? ?/sec 1.00 1622.2±125.58µs ? ?/sec
> physical_join_consider_sort 1.02 2.3±0.05ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
> physical_join_distinct 1.01 356.9±8.17µs ? ?/sec 1.00 353.0±3.93µs ? ?/sec
> physical_many_self_joins 1.01 12.7±0.08ms ? ?/sec 1.00 12.5±0.26ms ? ?/sec
> physical_plan_clickbench_all 1.01 203.2±1.51ms ? ?/sec 1.00 200.9±1.71ms ? ?/sec
> physical_plan_clickbench_q1 1.01 2.2±0.02ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
> physical_plan_clickbench_q10 1.05 3.8±0.14ms ? ?/sec 1.00 3.7±0.10ms ? ?/sec
> physical_plan_clickbench_q11 1.04 4.3±0.09ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
> physical_plan_clickbench_q12 1.05 4.5±0.09ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
> physical_plan_clickbench_q13 1.06 4.0±0.06ms ? ?/sec 1.00 3.8±0.12ms ? ?/sec
> physical_plan_clickbench_q14 1.05 4.3±0.09ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
> physical_plan_clickbench_q15 1.04 4.0±0.07ms ? ?/sec 1.00 3.9±0.09ms ? ?/sec
> physical_plan_clickbench_q16 1.05 3.9±0.07ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
> physical_plan_clickbench_q17 1.05 4.0±0.07ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
> physical_plan_clickbench_q18 1.02 2.7±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
> physical_plan_clickbench_q19 1.02 4.3±0.12ms ? ?/sec 1.00 4.2±0.09ms ? ?/sec
> physical_plan_clickbench_q2 1.03 2.9±0.03ms ? ?/sec 1.00 2.8±0.06ms ? ?/sec
> physical_plan_clickbench_q20 1.01 2.2±0.03ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
> physical_plan_clickbench_q21 1.01 2.8±0.04ms ? ?/sec 1.00 2.8±0.05ms ? ?/sec
> physical_plan_clickbench_q22 1.04 4.1±0.07ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
> physical_plan_clickbench_q23 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
> physical_plan_clickbench_q24 1.02 4.9±0.07ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
> physical_plan_clickbench_q25 1.01 3.5±0.03ms ? ?/sec 1.00 3.5±0.09ms ? ?/sec
> physical_plan_clickbench_q26 1.02 3.0±0.06ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
> physical_plan_clickbench_q27 1.01 3.6±0.10ms ? ?/sec 1.00 3.5±0.06ms ? ?/sec
> physical_plan_clickbench_q28 1.02 4.5±0.10ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
> physical_plan_clickbench_q29 1.02 4.8±0.07ms ? ?/sec 1.00 4.7±0.04ms ? ?/sec
> physical_plan_clickbench_q3 1.02 2.6±0.03ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
> physical_plan_clickbench_q30 1.04 16.1±0.21ms ? ?/sec 1.00 15.5±0.22ms ? ?/sec
> physical_plan_clickbench_q31 1.02 4.5±0.03ms ? ?/sec 1.00 4.5±0.05ms ? ?/sec
> physical_plan_clickbench_q32 1.02 4.5±0.04ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
> physical_plan_clickbench_q33 1.02 3.7±0.09ms ? ?/sec 1.00 3.6±0.08ms ? ?/sec
> physical_plan_clickbench_q34 1.02 3.3±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
> physical_plan_clickbench_q35 1.01 3.4±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
> physical_plan_clickbench_q36 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
> physical_plan_clickbench_q37 1.02 4.7±0.09ms ? ?/sec 1.00 4.7±0.06ms ? ?/sec
> physical_plan_clickbench_q38 1.01 4.7±0.04ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
> physical_plan_clickbench_q39 1.01 4.1±0.07ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
> physical_plan_clickbench_q4 1.03 2.2±0.04ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
> physical_plan_clickbench_q40 1.01 5.0±0.05ms ? ?/sec 1.00 4.9±0.07ms ? ?/sec
> physical_plan_clickbench_q41 1.01 4.3±0.03ms ? ?/sec 1.00 4.3±0.08ms ? ?/sec
> physical_plan_clickbench_q42 1.01 4.3±0.07ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
> physical_plan_clickbench_q43 1.01 4.6±0.06ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
> physical_plan_clickbench_q44 1.01 2.3±0.08ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
> physical_plan_clickbench_q45 1.01 2.3±0.02ms ? ?/sec 1.00 2.3±0.03ms ? ?/sec
> physical_plan_clickbench_q46 1.01 3.2±0.08ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
> physical_plan_clickbench_q47 1.02 4.8±0.05ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
> physical_plan_clickbench_q48 1.00 5.2±0.06ms ? ?/sec 1.00 5.3±0.19ms ? ?/sec
> physical_plan_clickbench_q49 1.01 5.5±0.11ms ? ?/sec 1.00 5.4±0.08ms ? ?/sec
> physical_plan_clickbench_q5 1.04 2.6±0.10ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
> physical_plan_clickbench_q50 1.01 4.2±0.11ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
> physical_plan_clickbench_q51 1.01 3.6±0.08ms ? ?/sec 1.00 3.6±0.07ms ? ?/sec
> physical_plan_clickbench_q6 1.05 2.6±0.07ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
> physical_plan_clickbench_q7 1.04 2.2±0.04ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
> physical_plan_clickbench_q8 1.05 3.6±0.10ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
> physical_plan_clickbench_q9 1.04 3.8±0.09ms ? ?/sec 1.00 3.6±0.08ms ? ?/sec
> physical_plan_tpcds_all 1.00 1968.6±20.06ms ? ?/sec 1.00 1966.2±18.94ms ? ?/sec
> physical_plan_tpch_all 1.00 129.4±1.21ms ? ?/sec 1.00 129.9±1.06ms ? ?/sec
> physical_plan_tpch_q1 1.03 3.1±0.03ms ? ?/sec 1.00 3.0±0.06ms ? ?/sec
> physical_plan_tpch_q10 1.01 7.4±0.06ms ? ?/sec 1.00 7.3±0.10ms ? ?/sec
> physical_plan_tpch_q11 1.01 8.7±0.15ms ? ?/sec 1.00 8.6±0.17ms ? ?/sec
> physical_plan_tpch_q12 1.01 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
> physical_plan_tpch_q13 1.01 3.1±0.06ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
> physical_plan_tpch_q14 1.02 3.2±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
> physical_plan_tpch_q16 1.01 5.3±0.04ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
> physical_plan_tpch_q17 1.03 5.8±0.04ms ? ?/sec 1.00 5.6±0.07ms ? ?/sec
> physical_plan_tpch_q18 1.01 6.1±0.06ms ? ?/sec 1.00 6.0±0.10ms ? ?/sec
> physical_plan_tpch_q19 1.03 5.3±0.03ms ? ?/sec 1.00 5.1±0.10ms ? ?/sec
> physical_plan_tpch_q2 1.03 12.7±0.13ms ? ?/sec 1.00 12.4±0.17ms ? ?/sec
> physical_plan_tpch_q20 1.00 8.2±0.06ms ? ?/sec 1.01 8.3±0.15ms ? ?/sec
> physical_plan_tpch_q21 1.01 10.3±0.10ms ? ?/sec 1.00 10.2±0.14ms ? ?/sec
> physical_plan_tpch_q22 1.01 6.6±0.10ms ? ?/sec 1.00 6.5±0.06ms ? ?/sec
> physical_plan_tpch_q3 1.02 5.7±0.04ms ? ?/sec 1.00 5.6±0.04ms ? ?/sec
> physical_plan_tpch_q4 1.01 3.1±0.05ms ? ?/sec 1.00 3.0±0.07ms ? ?/sec
> physical_plan_tpch_q5 1.01 6.0±0.04ms ? ?/sec 1.00 6.0±0.10ms ? ?/sec
> physical_plan_tpch_q6 1.01 1617.3±16.07µs ? ?/sec 1.00 1604.1±25.37µs ? ?/sec
> physical_plan_tpch_q7 1.02 7.4±0.08ms ? ?/sec 1.00 7.2±0.12ms ? ?/sec
> physical_plan_tpch_q8 1.01 9.5±0.07ms ? ?/sec 1.00 9.3±0.09ms ? ?/sec
> physical_plan_tpch_q9 1.00 6.8±0.04ms ? ?/sec 1.00 6.8±0.10ms ? ?/sec
> physical_select_aggregates_from_200 1.00 17.7±0.13ms ? ?/sec 1.00 17.7±0.11ms ? ?/sec
> physical_select_all_from_1000 1.00 23.6±0.24ms ? ?/sec 1.05 24.7±0.17ms ? ?/sec
> physical_select_one_from_700 1.02 1357.1±8.44µs ? ?/sec 1.00 1327.1±11.77µs ? ?/sec
> physical_sorted_union_order_by_10_int64 1.02 11.3±0.09ms ? ?/sec 1.00 11.1±0.11ms ? ?/sec
> physical_sorted_union_order_by_10_uint64 1.02 30.8±0.37ms ? ?/sec 1.00 30.2±0.37ms ? ?/sec
> physical_sorted_union_order_by_50_int64 1.01 200.9±2.34ms ? ?/sec 1.00 199.8±4.52ms ? ?/sec
> physical_sorted_union_order_by_50_uint64 1.04 1148.2±7.76ms ? ?/sec 1.00 1104.5±15.61ms ? ?/sec
> physical_theta_join_consider_sort 1.07 2.8±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
> physical_unnest_to_join 1.05 3.2±0.04ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
> physical_window_function_partition_by_12_on_values 1.02 1612.1±18.59µs ? ?/sec 1.00 1587.3±11.05µs ? ?/sec
> physical_window_function_partition_by_30_on_values 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.06ms ? ?/sec
> physical_window_function_partition_by_4_on_values 1.01 1095.9±10.86µs ? ?/sec 1.00 1085.1±20.62µs ? ?/sec
> physical_window_function_partition_by_7_on_values 1.02 1280.3±30.33µs ? ?/sec 1.00 1261.3±17.40µs ? ?/sec
> physical_window_function_partition_by_8_on_values 1.02 1347.7±7.20µs ? ?/sec 1.00 1323.7±12.37µs ? ?/sec
> with_param_values_many_columns 1.00 577.1±5.70µs ? ?/sec 1.05 606.5±14.52µs ? ?/sec
> ```
@alamb are there any of these that use structs? It seems like this has no impact on the benchmarks (good!) but maybe we should add some that hit the full rewrite?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843805436
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing get-field-pushdown-try-3 (e95acd3972b0ef198c461aa706c091830f9b9bc3) to 9962911ee5d5cdde5f76a4489ee7d432365dde0c [diff](https://github.com/apache/datafusion/compare/9962911ee5d5cdde5f76a4489ee7d432365dde0c..e95acd3972b0ef198c461aa706c091830f9b9bc3)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=get-field-pushdown-try-3
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893435394
😬 not a real fan of the slowdown in planning (looks like 1-3% across the board)
I think we can probably get it back by the tiem 53 ships with some various other optimziations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "ethan-tyler (via GitHub)" <gi...@apache.org>.
ethan-tyler commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2801417570
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -95,9 +101,533 @@ impl OptimizerRule for ExtractLeafExpressions {
fn rewrite(
&self,
plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
+ config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
- Ok(Transformed::no(plan))
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// Works for any number of inputs (0, 1, 2, …N). For multi-input nodes
+/// like Join, each extracted sub-expression is routed to the correct input
+/// by checking which input's schema contains all of the expression's column
+/// references.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Only extract from plan types whose output schema is predictable after
+ // expression rewriting. Nodes like Window derive column names from
+ // their expressions, so rewriting `get_field` inside a window function
+ // changes the output schema and breaks the recovery projection.
+ if !matches!(
+ &plan,
+ LogicalPlan::Aggregate(_)
+ | LogicalPlan::Filter(_)
+ | LogicalPlan::Sort(_)
+ | LogicalPlan::Limit(_)
+ | LogicalPlan::Join(_)
+ ) {
+ return Ok(Transformed::no(plan));
+ }
+
+ let inputs = plan.inputs();
+ if inputs.is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ // Save original output schema before any transformation
+ let original_schema = Arc::clone(plan.schema());
+
+ // Build per-input schemas from borrowed inputs (before plan is consumed
+ // by map_expressions). We only need schemas and column sets for routing;
+ // the actual inputs are cloned later only if extraction succeeds.
+ let input_schemas: Vec<Arc<DFSchema>> =
+ inputs.iter().map(|i| Arc::clone(i.schema())).collect();
+
+ // Build per-input extractors
+ let mut extractors: Vec<LeafExpressionExtractor> = input_schemas
+ .iter()
+ .map(|schema| LeafExpressionExtractor::new(schema.as_ref(), alias_generator))
+ .collect();
+
+ // Build per-input column sets for routing expressions to the correct input
+ let input_column_sets: Vec<std::collections::HashSet<Column>> = input_schemas
+ .iter()
+ .map(|schema| schema_columns(schema.as_ref()))
+ .collect();
+
+ // Transform expressions via map_expressions with routing
+ let transformed = plan.map_expressions(|expr| {
+ routing_extract(expr, &mut extractors, &input_column_sets)
+ })?;
+
+ // If no expressions were rewritten, nothing was extracted
+ if !transformed.transformed {
+ return Ok(transformed);
+ }
+
+ // Clone inputs now that we know extraction succeeded. Wrap in Arc
+ // upfront since build_extraction_projection expects &Arc<LogicalPlan>.
+ let owned_inputs: Vec<Arc<LogicalPlan>> = transformed
+ .data
+ .inputs()
+ .into_iter()
+ .map(|i| Arc::new(i.clone()))
+ .collect();
+
+ // Build per-input extraction projections (None means no extractions for that input)
+ let new_inputs: Vec<LogicalPlan> = owned_inputs
+ .into_iter()
+ .zip(extractors.iter())
+ .map(|(input_arc, extractor)| {
+ match extractor.build_extraction_projection(&input_arc)? {
+ Some(plan) => Ok(plan),
+ // No extractions for this input — recover the LogicalPlan
+ // without cloning (refcount is 1 since build returned None).
+ None => {
+ Ok(Arc::try_unwrap(input_arc).unwrap_or_else(|arc| (*arc).clone()))
+ }
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // Rebuild the plan keeping its rewritten expressions but replacing
+ // inputs with the new extraction projections.
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Add recovery projection if the output schema changed
+ let recovered = build_recovery_projection(original_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Given an expression, returns the index of the input whose columns fully
+/// cover the expression's column references.
+/// Returns `None` if the expression references columns from multiple inputs
+/// or if multiple inputs match (ambiguous, e.g. unqualified columns present
+/// in both sides of a join).
+fn find_owning_input(
+ expr: &Expr,
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Option<usize> {
+ let mut found = None;
+ for (idx, cols) in input_column_sets.iter().enumerate() {
+ if has_all_column_refs(expr, cols) {
+ if found.is_some() {
+ // Ambiguous — multiple inputs match
+ return None;
+ }
+ found = Some(idx);
+ }
+ }
+ found
+}
+
+/// Walks an expression tree top-down, extracting `MoveTowardsLeafNodes`
+/// sub-expressions and routing each to the correct per-input extractor.
+fn routing_extract(
+ expr: Expr,
+ extractors: &mut [LeafExpressionExtractor],
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Result<Transformed<Expr>> {
+ expr.transform_down(|e| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &e
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: e,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
Review Comment:
Totally - just wanted to call it out. The comment was a good choice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893399227
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group get-field-pushdown-try-3 main
----- ------------------------ ----
logical_aggregate_with_join 1.00 631.6±5.78µs ? ?/sec 1.00 630.2±5.94µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 290.9±3.56µs ? ?/sec 1.01 292.5±6.25µs ? ?/sec
logical_select_all_from_1000 1.00 10.5±0.04ms ? ?/sec 1.01 10.6±0.09ms ? ?/sec
logical_select_one_from_700 1.00 414.3±5.17µs ? ?/sec 1.00 414.6±6.22µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 371.9±1.42µs ? ?/sec 1.00 370.5±2.26µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 359.7±2.80µs ? ?/sec 1.00 358.8±11.72µs ? ?/sec
physical_intersection 1.05 1708.0±29.65µs ? ?/sec 1.00 1628.0±57.53µs ? ?/sec
physical_join_consider_sort 1.12 2.5±0.05ms ? ?/sec 1.00 2.3±0.03ms ? ?/sec
physical_join_distinct 1.00 347.0±1.69µs ? ?/sec 1.01 352.1±15.29µs ? ?/sec
physical_many_self_joins 1.14 14.3±0.29ms ? ?/sec 1.00 12.5±0.12ms ? ?/sec
physical_plan_clickbench_all 1.05 212.2±3.07ms ? ?/sec 1.00 202.6±3.44ms ? ?/sec
physical_plan_clickbench_q1 1.02 2.2±0.03ms ? ?/sec 1.00 2.1±0.03ms ? ?/sec
physical_plan_clickbench_q10 1.03 3.8±0.06ms ? ?/sec 1.00 3.7±0.09ms ? ?/sec
physical_plan_clickbench_q11 1.04 4.3±0.09ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q12 1.05 4.5±0.09ms ? ?/sec 1.00 4.3±0.08ms ? ?/sec
physical_plan_clickbench_q13 1.05 4.0±0.13ms ? ?/sec 1.00 3.8±0.12ms ? ?/sec
physical_plan_clickbench_q14 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.14ms ? ?/sec
physical_plan_clickbench_q15 1.04 4.0±0.08ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q16 1.04 3.8±0.10ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.04 3.9±0.10ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.04 2.8±0.05ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.05 4.4±0.12ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q2 1.03 2.9±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q20 1.05 2.3±0.03ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.04 2.9±0.02ms ? ?/sec 1.00 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q22 1.05 4.1±0.08ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q23 1.06 4.5±0.11ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.08 5.3±0.07ms ? ?/sec 1.00 4.9±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.05 3.7±0.11ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.04 3.0±0.04ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.03 3.6±0.05ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q28 1.07 4.8±0.12ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q29 1.08 5.1±0.10ms ? ?/sec 1.00 4.8±0.07ms ? ?/sec
physical_plan_clickbench_q3 1.04 2.6±0.05ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.09 17.2±0.19ms ? ?/sec 1.00 15.8±0.26ms ? ?/sec
physical_plan_clickbench_q31 1.06 4.7±0.06ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q32 1.07 4.7±0.06ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.05 3.8±0.06ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
physical_plan_clickbench_q34 1.05 3.4±0.10ms ? ?/sec 1.00 3.2±0.05ms ? ?/sec
physical_plan_clickbench_q35 1.05 3.5±0.09ms ? ?/sec 1.00 3.4±0.05ms ? ?/sec
physical_plan_clickbench_q36 1.07 4.5±0.14ms ? ?/sec 1.00 4.2±0.09ms ? ?/sec
physical_plan_clickbench_q37 1.09 5.1±0.10ms ? ?/sec 1.00 4.7±0.12ms ? ?/sec
physical_plan_clickbench_q38 1.07 5.0±0.09ms ? ?/sec 1.00 4.7±0.06ms ? ?/sec
physical_plan_clickbench_q39 1.06 4.3±0.06ms ? ?/sec 1.00 4.1±0.06ms ? ?/sec
physical_plan_clickbench_q4 1.03 2.2±0.02ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.07 5.3±0.09ms ? ?/sec 1.00 4.9±0.08ms ? ?/sec
physical_plan_clickbench_q41 1.07 4.6±0.06ms ? ?/sec 1.00 4.3±0.10ms ? ?/sec
physical_plan_clickbench_q42 1.06 4.5±0.08ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q43 1.05 4.8±0.07ms ? ?/sec 1.00 4.6±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.06 2.4±0.02ms ? ?/sec 1.00 2.3±0.04ms ? ?/sec
physical_plan_clickbench_q45 1.04 2.4±0.02ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.05 3.4±0.06ms ? ?/sec 1.00 3.2±0.05ms ? ?/sec
physical_plan_clickbench_q47 1.06 5.1±0.10ms ? ?/sec 1.00 4.8±0.12ms ? ?/sec
physical_plan_clickbench_q48 1.06 5.5±0.08ms ? ?/sec 1.00 5.2±0.08ms ? ?/sec
physical_plan_clickbench_q49 1.05 5.8±0.10ms ? ?/sec 1.00 5.5±0.14ms ? ?/sec
physical_plan_clickbench_q5 1.04 2.6±0.05ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q50 1.07 4.4±0.08ms ? ?/sec 1.00 4.1±0.06ms ? ?/sec
physical_plan_clickbench_q51 1.05 3.7±0.07ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.03 2.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.04 2.1±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_clickbench_q8 1.03 3.6±0.04ms ? ?/sec 1.00 3.5±0.07ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.7±0.03ms ? ?/sec 1.00 3.6±0.16ms ? ?/sec
physical_plan_struct_join_agg_sort 1.26 3.4±0.04ms ? ?/sec 1.00 2.7±0.11ms ? ?/sec
physical_plan_tpcds_all 1.05 2.0±0.02s ? ?/sec 1.00 1926.0±17.16ms ? ?/sec
physical_plan_tpch_all 1.06 134.1±4.47ms ? ?/sec 1.00 127.0±1.79ms ? ?/sec
physical_plan_tpch_q1 1.05 3.1±0.06ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q10 1.05 7.6±0.06ms ? ?/sec 1.00 7.2±0.11ms ? ?/sec
physical_plan_tpch_q11 1.03 8.9±0.16ms ? ?/sec 1.00 8.6±0.31ms ? ?/sec
physical_plan_tpch_q12 1.03 3.2±0.08ms ? ?/sec 1.00 3.1±0.10ms ? ?/sec
physical_plan_tpch_q13 1.03 3.1±0.02ms ? ?/sec 1.00 3.0±0.06ms ? ?/sec
physical_plan_tpch_q14 1.05 3.2±0.05ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_tpch_q16 1.05 5.4±0.20ms ? ?/sec 1.00 5.2±0.04ms ? ?/sec
physical_plan_tpch_q17 1.04 5.8±0.10ms ? ?/sec 1.00 5.6±0.15ms ? ?/sec
physical_plan_tpch_q18 1.04 6.2±0.09ms ? ?/sec 1.00 5.9±0.07ms ? ?/sec
physical_plan_tpch_q19 1.06 5.3±0.09ms ? ?/sec 1.00 5.0±0.06ms ? ?/sec
physical_plan_tpch_q2 1.03 12.8±0.12ms ? ?/sec 1.00 12.4±0.13ms ? ?/sec
physical_plan_tpch_q20 1.02 8.3±0.19ms ? ?/sec 1.00 8.1±0.09ms ? ?/sec
physical_plan_tpch_q21 1.06 10.7±0.41ms ? ?/sec 1.00 10.1±0.15ms ? ?/sec
physical_plan_tpch_q22 1.04 6.7±0.25ms ? ?/sec 1.00 6.5±0.09ms ? ?/sec
physical_plan_tpch_q3 1.04 5.8±0.09ms ? ?/sec 1.00 5.6±0.03ms ? ?/sec
physical_plan_tpch_q4 1.04 3.1±0.04ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q5 1.04 6.2±0.04ms ? ?/sec 1.00 6.0±0.06ms ? ?/sec
physical_plan_tpch_q6 1.05 1671.5±30.09µs ? ?/sec 1.00 1594.1±20.41µs ? ?/sec
physical_plan_tpch_q7 1.04 7.4±0.05ms ? ?/sec 1.00 7.2±0.06ms ? ?/sec
physical_plan_tpch_q8 1.04 9.7±0.11ms ? ?/sec 1.00 9.3±0.13ms ? ?/sec
physical_plan_tpch_q9 1.04 6.9±0.06ms ? ?/sec 1.00 6.7±0.13ms ? ?/sec
physical_select_aggregates_from_200 1.34 23.5±0.16ms ? ?/sec 1.00 17.5±0.23ms ? ?/sec
physical_select_all_from_1000 1.03 24.2±0.16ms ? ?/sec 1.00 23.5±0.20ms ? ?/sec
physical_select_one_from_700 1.14 1504.1±16.23µs ? ?/sec 1.00 1319.4±7.33µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.02 11.3±0.06ms ? ?/sec 1.00 11.1±0.08ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.03 31.2±0.53ms ? ?/sec 1.00 30.3±1.06ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.03 203.8±3.01ms ? ?/sec 1.00 197.8±2.15ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.08 1178.0±11.09ms ? ?/sec 1.00 1092.7±11.37ms ? ?/sec
physical_theta_join_consider_sort 1.10 2.9±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_unnest_to_join 1.12 3.4±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.02 1601.0±22.79µs ? ?/sec 1.00 1569.2±15.66µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.01 3.0±0.02ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.02 1105.6±20.22µs ? ?/sec 1.00 1080.6±8.74µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.02 1275.1±12.23µs ? ?/sec 1.00 1253.2±13.27µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.02 1342.5±9.41µs ? ?/sec 1.00 1321.6±18.03µs ? ?/sec
with_param_values_many_columns 1.00 580.3±5.03µs ? ?/sec 1.01 587.0±5.30µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893436142
🤖 Hi @alamb, thanks for the request (https://github.com/apache/datafusion/pull/20117#issuecomment-3893435763).
[`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
- Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
- Criterion: aggregate_query_sql, aggregate_vectorized, case_when, character_length, in_list, left, range_and_generate_series, reset_plan_states, sort, sql_planner, strpos, substr_index, with_hashes
Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`
You can also set environment variables on subsequent lines:
```
run benchmark tpch_mem
DATAFUSION_RUNTIME_MEMORY_LIMIT=1G
```
Unsupported benchmarks: sql_planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756607559
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
Review Comment:
I'm not sure what else we could handle here. Maybe Extension?
Possibly before we merge this PR we expand this to explicitly ignore all other nodes so that if a new node is added one has to decide how this rule should handle it.
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing extracted projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild the path from target back up to our node's input
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
+ .chain(
+ transformed
+ .data
+ .inputs()
+ .iter()
+ .skip(1)
+ .map(|p| (*p).clone()),
+ )
+ .collect();
+
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Use CSE's pattern: add recovery projection to restore original schema
+ let recovered = build_recover_project_plan(input_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Join nodes.
+///
+/// For Joins, we extract from:
+/// - `on` expressions: pairs of (left_key, right_key) for equijoin
+/// - `filter` expression: non-equi join conditions
+///
+/// Each expression is routed to the appropriate side (left or right) based on
+/// which columns it references. Expressions referencing columns from both sides
+/// cannot have sub-expressions extracted (they must remain in the filter).
+fn extract_from_join(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Join(join) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ let left_schema = join.left.schema();
+ let right_schema = join.right.schema();
+
+ // Create extractors for left and right sides
+ // Find extraction targets for each side (look through schema-preserving nodes)
+ let (left_target, left_path) = find_extraction_target(&join.left);
+ let (right_target, right_path) = find_extraction_target(&join.right);
+
+ let left_target_schema = Arc::clone(left_target.schema());
+ let right_target_schema = Arc::clone(right_target.schema());
+
+ let mut left_extractor =
+ LeafExpressionExtractor::new(left_target_schema.as_ref(), alias_generator);
+ let mut right_extractor =
+ LeafExpressionExtractor::new(right_target_schema.as_ref(), alias_generator);
+
+ // Build column checker to route expressions to correct side
+ let mut column_checker = ColumnChecker::new(left_schema.as_ref(), right_schema.as_ref());
+
+ // Extract from `on` expressions (equijoin keys)
+ let mut new_on = Vec::with_capacity(join.on.len());
+ let mut any_extracted = false;
+
+ for (left_key, right_key) in &join.on {
+ // Left key should reference only left columns
+ let new_left = left_extractor.extract(left_key.clone())?;
+ if new_left.transformed {
+ any_extracted = true;
+ }
+
+ // Right key should reference only right columns
+ let new_right = right_extractor.extract(right_key.clone())?;
+ if new_right.transformed {
+ any_extracted = true;
+ }
+
+ new_on.push((new_left.data, new_right.data));
+ }
+
+ // Extract from `filter` expression
+ let new_filter = if let Some(ref filter) = join.filter {
+ let extracted = extract_from_join_filter(
+ filter.clone(),
+ &mut column_checker,
+ &mut left_extractor,
+ &mut right_extractor,
+ )?;
+ if extracted.transformed {
+ any_extracted = true;
+ }
+ Some(extracted.data)
+ } else {
+ None
+ };
+
+ if !any_extracted {
+ return Ok(Transformed::no(LogicalPlan::Join(join)));
+ }
+
+ // Save original schema before modifying inputs
+ let original_schema = Arc::clone(&join.schema);
+
+ // Build left extraction projection if needed
+ let new_left = if left_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&left_target) {
+ merge_into_extracted_projection(existing_proj, &left_extractor)?
+ } else {
+ left_extractor.build_projection_with_all_columns(left_target)?
+ };
+ Arc::new(rebuild_path(left_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.left)
+ };
+
+ // Build right extraction projection if needed
+ let new_right = if right_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&right_target) {
+ merge_into_extracted_projection(existing_proj, &right_extractor)?
+ } else {
+ right_extractor.build_projection_with_all_columns(right_target)?
+ };
+ Arc::new(rebuild_path(right_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.right)
+ };
+
+ // Create new Join with updated inputs and expressions
+ let new_join = datafusion_expr::logical_plan::Join::try_new(
+ new_left,
+ new_right,
+ new_on,
+ new_filter,
+ join.join_type,
+ join.join_constraint,
+ join.null_equality,
+ join.null_aware,
+ )?;
+
+ // Add recovery projection to restore original schema
+ // This hides the intermediate extracted expression columns
+ let recovered = build_recover_project_plan(original_schema.as_ref(), LogicalPlan::Join(new_join))?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a join filter expression.
+///
+/// For each sub-expression, determines if it references only left, only right,
+/// or both columns, and routes extractions accordingly.
+fn extract_from_join_filter(
+ filter: Expr,
+ column_checker: &mut ColumnChecker,
+ left_extractor: &mut LeafExpressionExtractor,
+ right_extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ filter.transform_down(|expr| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: expr,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
+
+ match expr.placement() {
+ ExpressionPlacement::MoveTowardsLeafNodes => {
+ // Check which side this expression belongs to
+ if column_checker.is_left_only(&expr) {
+ // Extract to left side
+ let col_ref = left_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else if column_checker.is_right_only(&expr) {
+ // Extract to right side
+ let col_ref = right_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else {
+ // References both sides - cannot extract, keep in place
+ // This shouldn't typically happen for MoveTowardsLeafNodes expressions
+ // but we handle it gracefully
+ Ok(Transformed::no(expr))
+ }
+ }
+ ExpressionPlacement::Column => {
+ // Track columns for pass-through on appropriate side
+ if let Expr::Column(col) = &expr {
+ if column_checker.is_left_only(&expr) {
+ left_extractor.columns_needed.insert(col.clone());
+ } else if column_checker.is_right_only(&expr) {
+ right_extractor.columns_needed.insert(col.clone());
+ }
+ }
+ Ok(Transformed::no(expr))
+ }
+ _ => {
+ // Continue recursing into children
+ Ok(Transformed::no(expr))
+ }
+ }
+ })
+}
+
+/// Evaluates the columns referenced in the given expression to see if they refer
+/// only to the left or right columns of a join.
+struct ColumnChecker<'a> {
+ left_schema: &'a DFSchema,
+ left_columns: Option<std::collections::HashSet<Column>>,
+ right_schema: &'a DFSchema,
+ right_columns: Option<std::collections::HashSet<Column>>,
+}
+
+impl<'a> ColumnChecker<'a> {
+ fn new(left_schema: &'a DFSchema, right_schema: &'a DFSchema) -> Self {
+ Self {
+ left_schema,
+ left_columns: None,
+ right_schema,
+ right_columns: None,
+ }
+ }
+
+ /// Return true if the expression references only columns from the left side
+ fn is_left_only(&mut self, predicate: &Expr) -> bool {
+ if self.left_columns.is_none() {
+ self.left_columns = Some(schema_columns(self.left_schema));
+ }
+ has_all_column_refs(predicate, self.left_columns.as_ref().unwrap())
+ }
+
+ /// Return true if the expression references only columns from the right side
+ fn is_right_only(&mut self, predicate: &Expr) -> bool {
+ if self.right_columns.is_none() {
+ self.right_columns = Some(schema_columns(self.right_schema));
+ }
+ has_all_column_refs(predicate, self.right_columns.as_ref().unwrap())
+ }
+}
+
+/// Returns all columns in the schema (both qualified and unqualified forms)
+fn schema_columns(schema: &DFSchema) -> std::collections::HashSet<Column> {
+ schema
+ .iter()
+ .flat_map(|(qualifier, field)| {
+ [
+ Column::new(qualifier.cloned(), field.name()),
+ Column::new_unqualified(field.name()),
+ ]
+ })
+ .collect()
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Aggregate nodes.
+///
+/// For Aggregates, we extract from:
+/// - Group-by expressions (full expressions or sub-expressions)
+/// - Arguments inside aggregate functions (NOT the aggregate function itself)
+///
+/// Uses CSE's two-level pattern with NamePreserver for stable name handling.
+fn extract_from_aggregate(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Aggregate(agg) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_group_names: Vec<_> = agg
+ .group_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+ let saved_aggr_names: Vec<_> = agg
+ .aggr_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+
+ // Find where to place extractions
+ let (target, path) = find_extraction_target(&agg.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from group-by expressions
+ let mut new_group_by = Vec::with_capacity(agg.group_expr.len());
+ let mut has_extractions = false;
+
+ for expr in &agg.group_expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_group_by.push(transformed.data);
+ }
+
+ // Extract from aggregate function arguments (not the function itself)
+ let mut new_aggr = Vec::with_capacity(agg.aggr_expr.len());
+
+ for expr in &agg.aggr_expr {
+ let transformed = extract_from_aggregate_args(expr.clone(), &mut extractor)?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_aggr.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Aggregate(agg)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Restore names in group-by expressions using NamePreserver
+ let restored_group_expr: Vec<Expr> = new_group_by
+ .into_iter()
+ .zip(saved_group_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Restore names in aggregate expressions using NamePreserver
+ let restored_aggr_expr: Vec<Expr> = new_aggr
+ .into_iter()
+ .zip(saved_aggr_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Create new Aggregate with restored names
+ // (no outer projection needed if names are properly preserved)
+ let new_agg = datafusion_expr::logical_plan::Aggregate::try_new(
+ Arc::new(rebuilt_input),
+ restored_group_expr,
+ restored_aggr_expr,
+ )?;
+
+ Ok(Transformed::yes(LogicalPlan::Aggregate(new_agg)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Projection nodes.
+///
+/// Uses CSE's two-level pattern (outer + inner projections only):
+/// - Inner projection: extraction with ALL columns passed through
+/// - Outer projection: rewritten expressions with restored names
+///
+/// This avoids the unstable 3-level structure that gets broken by OptimizeProjections.
+fn extract_from_projection(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Projection(proj) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Skip if this projection is fully extracted (only column references)
+ if is_fully_extracted(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
Review Comment:
I think this should skip extraction on any expressions that has all `ExprPlacement::should_push_to_leaves` (i.e. also include columns). E.g. if we have `[id, get_field(s, 'foo')]` we can push down the entire projection without extraction.
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing extracted projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild the path from target back up to our node's input
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
+ .chain(
+ transformed
+ .data
+ .inputs()
+ .iter()
+ .skip(1)
+ .map(|p| (*p).clone()),
+ )
+ .collect();
+
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Use CSE's pattern: add recovery projection to restore original schema
+ let recovered = build_recover_project_plan(input_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Join nodes.
+///
+/// For Joins, we extract from:
+/// - `on` expressions: pairs of (left_key, right_key) for equijoin
+/// - `filter` expression: non-equi join conditions
+///
+/// Each expression is routed to the appropriate side (left or right) based on
+/// which columns it references. Expressions referencing columns from both sides
+/// cannot have sub-expressions extracted (they must remain in the filter).
+fn extract_from_join(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Join(join) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ let left_schema = join.left.schema();
+ let right_schema = join.right.schema();
+
+ // Create extractors for left and right sides
+ // Find extraction targets for each side (look through schema-preserving nodes)
+ let (left_target, left_path) = find_extraction_target(&join.left);
+ let (right_target, right_path) = find_extraction_target(&join.right);
+
+ let left_target_schema = Arc::clone(left_target.schema());
+ let right_target_schema = Arc::clone(right_target.schema());
+
+ let mut left_extractor =
+ LeafExpressionExtractor::new(left_target_schema.as_ref(), alias_generator);
+ let mut right_extractor =
+ LeafExpressionExtractor::new(right_target_schema.as_ref(), alias_generator);
+
+ // Build column checker to route expressions to correct side
+ let mut column_checker = ColumnChecker::new(left_schema.as_ref(), right_schema.as_ref());
+
+ // Extract from `on` expressions (equijoin keys)
+ let mut new_on = Vec::with_capacity(join.on.len());
+ let mut any_extracted = false;
+
+ for (left_key, right_key) in &join.on {
+ // Left key should reference only left columns
+ let new_left = left_extractor.extract(left_key.clone())?;
+ if new_left.transformed {
+ any_extracted = true;
+ }
+
+ // Right key should reference only right columns
+ let new_right = right_extractor.extract(right_key.clone())?;
+ if new_right.transformed {
+ any_extracted = true;
+ }
+
+ new_on.push((new_left.data, new_right.data));
+ }
+
+ // Extract from `filter` expression
+ let new_filter = if let Some(ref filter) = join.filter {
+ let extracted = extract_from_join_filter(
+ filter.clone(),
+ &mut column_checker,
+ &mut left_extractor,
+ &mut right_extractor,
+ )?;
+ if extracted.transformed {
+ any_extracted = true;
+ }
+ Some(extracted.data)
+ } else {
+ None
+ };
+
+ if !any_extracted {
+ return Ok(Transformed::no(LogicalPlan::Join(join)));
+ }
+
+ // Save original schema before modifying inputs
+ let original_schema = Arc::clone(&join.schema);
+
+ // Build left extraction projection if needed
+ let new_left = if left_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&left_target) {
+ merge_into_extracted_projection(existing_proj, &left_extractor)?
+ } else {
+ left_extractor.build_projection_with_all_columns(left_target)?
+ };
+ Arc::new(rebuild_path(left_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.left)
+ };
+
+ // Build right extraction projection if needed
+ let new_right = if right_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&right_target) {
+ merge_into_extracted_projection(existing_proj, &right_extractor)?
+ } else {
+ right_extractor.build_projection_with_all_columns(right_target)?
+ };
+ Arc::new(rebuild_path(right_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.right)
+ };
+
+ // Create new Join with updated inputs and expressions
+ let new_join = datafusion_expr::logical_plan::Join::try_new(
+ new_left,
+ new_right,
+ new_on,
+ new_filter,
+ join.join_type,
+ join.join_constraint,
+ join.null_equality,
+ join.null_aware,
+ )?;
+
+ // Add recovery projection to restore original schema
+ // This hides the intermediate extracted expression columns
+ let recovered = build_recover_project_plan(original_schema.as_ref(), LogicalPlan::Join(new_join))?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a join filter expression.
+///
+/// For each sub-expression, determines if it references only left, only right,
+/// or both columns, and routes extractions accordingly.
+fn extract_from_join_filter(
+ filter: Expr,
+ column_checker: &mut ColumnChecker,
+ left_extractor: &mut LeafExpressionExtractor,
+ right_extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ filter.transform_down(|expr| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: expr,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
+
+ match expr.placement() {
+ ExpressionPlacement::MoveTowardsLeafNodes => {
+ // Check which side this expression belongs to
+ if column_checker.is_left_only(&expr) {
+ // Extract to left side
+ let col_ref = left_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else if column_checker.is_right_only(&expr) {
+ // Extract to right side
+ let col_ref = right_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else {
+ // References both sides - cannot extract, keep in place
+ // This shouldn't typically happen for MoveTowardsLeafNodes expressions
+ // but we handle it gracefully
+ Ok(Transformed::no(expr))
+ }
+ }
+ ExpressionPlacement::Column => {
+ // Track columns for pass-through on appropriate side
+ if let Expr::Column(col) = &expr {
+ if column_checker.is_left_only(&expr) {
+ left_extractor.columns_needed.insert(col.clone());
+ } else if column_checker.is_right_only(&expr) {
+ right_extractor.columns_needed.insert(col.clone());
+ }
+ }
+ Ok(Transformed::no(expr))
+ }
+ _ => {
+ // Continue recursing into children
+ Ok(Transformed::no(expr))
+ }
+ }
+ })
+}
+
+/// Evaluates the columns referenced in the given expression to see if they refer
+/// only to the left or right columns of a join.
+struct ColumnChecker<'a> {
+ left_schema: &'a DFSchema,
+ left_columns: Option<std::collections::HashSet<Column>>,
+ right_schema: &'a DFSchema,
+ right_columns: Option<std::collections::HashSet<Column>>,
+}
+
+impl<'a> ColumnChecker<'a> {
+ fn new(left_schema: &'a DFSchema, right_schema: &'a DFSchema) -> Self {
+ Self {
+ left_schema,
+ left_columns: None,
+ right_schema,
+ right_columns: None,
+ }
+ }
+
+ /// Return true if the expression references only columns from the left side
+ fn is_left_only(&mut self, predicate: &Expr) -> bool {
+ if self.left_columns.is_none() {
+ self.left_columns = Some(schema_columns(self.left_schema));
+ }
+ has_all_column_refs(predicate, self.left_columns.as_ref().unwrap())
+ }
+
+ /// Return true if the expression references only columns from the right side
+ fn is_right_only(&mut self, predicate: &Expr) -> bool {
+ if self.right_columns.is_none() {
+ self.right_columns = Some(schema_columns(self.right_schema));
+ }
+ has_all_column_refs(predicate, self.right_columns.as_ref().unwrap())
+ }
+}
+
+/// Returns all columns in the schema (both qualified and unqualified forms)
+fn schema_columns(schema: &DFSchema) -> std::collections::HashSet<Column> {
+ schema
+ .iter()
+ .flat_map(|(qualifier, field)| {
+ [
+ Column::new(qualifier.cloned(), field.name()),
+ Column::new_unqualified(field.name()),
+ ]
+ })
+ .collect()
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Aggregate nodes.
+///
+/// For Aggregates, we extract from:
+/// - Group-by expressions (full expressions or sub-expressions)
+/// - Arguments inside aggregate functions (NOT the aggregate function itself)
+///
+/// Uses CSE's two-level pattern with NamePreserver for stable name handling.
+fn extract_from_aggregate(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Aggregate(agg) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_group_names: Vec<_> = agg
+ .group_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+ let saved_aggr_names: Vec<_> = agg
+ .aggr_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+
+ // Find where to place extractions
+ let (target, path) = find_extraction_target(&agg.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from group-by expressions
+ let mut new_group_by = Vec::with_capacity(agg.group_expr.len());
+ let mut has_extractions = false;
+
+ for expr in &agg.group_expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_group_by.push(transformed.data);
+ }
+
+ // Extract from aggregate function arguments (not the function itself)
+ let mut new_aggr = Vec::with_capacity(agg.aggr_expr.len());
+
+ for expr in &agg.aggr_expr {
+ let transformed = extract_from_aggregate_args(expr.clone(), &mut extractor)?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_aggr.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Aggregate(agg)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Restore names in group-by expressions using NamePreserver
+ let restored_group_expr: Vec<Expr> = new_group_by
+ .into_iter()
+ .zip(saved_group_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Restore names in aggregate expressions using NamePreserver
+ let restored_aggr_expr: Vec<Expr> = new_aggr
+ .into_iter()
+ .zip(saved_aggr_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Create new Aggregate with restored names
+ // (no outer projection needed if names are properly preserved)
+ let new_agg = datafusion_expr::logical_plan::Aggregate::try_new(
+ Arc::new(rebuilt_input),
+ restored_group_expr,
+ restored_aggr_expr,
+ )?;
+
+ Ok(Transformed::yes(LogicalPlan::Aggregate(new_agg)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Projection nodes.
+///
+/// Uses CSE's two-level pattern (outer + inner projections only):
+/// - Inner projection: extraction with ALL columns passed through
+/// - Outer projection: rewritten expressions with restored names
+///
+/// This avoids the unstable 3-level structure that gets broken by OptimizeProjections.
+fn extract_from_projection(
Review Comment:
I think this needs substantial improvement
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing extracted projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild the path from target back up to our node's input
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
+ .chain(
+ transformed
+ .data
+ .inputs()
+ .iter()
+ .skip(1)
+ .map(|p| (*p).clone()),
+ )
+ .collect();
+
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Use CSE's pattern: add recovery projection to restore original schema
+ let recovered = build_recover_project_plan(input_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Join nodes.
+///
+/// For Joins, we extract from:
+/// - `on` expressions: pairs of (left_key, right_key) for equijoin
+/// - `filter` expression: non-equi join conditions
+///
+/// Each expression is routed to the appropriate side (left or right) based on
+/// which columns it references. Expressions referencing columns from both sides
+/// cannot have sub-expressions extracted (they must remain in the filter).
+fn extract_from_join(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Join(join) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ let left_schema = join.left.schema();
+ let right_schema = join.right.schema();
+
+ // Create extractors for left and right sides
+ // Find extraction targets for each side (look through schema-preserving nodes)
+ let (left_target, left_path) = find_extraction_target(&join.left);
+ let (right_target, right_path) = find_extraction_target(&join.right);
+
+ let left_target_schema = Arc::clone(left_target.schema());
+ let right_target_schema = Arc::clone(right_target.schema());
+
+ let mut left_extractor =
+ LeafExpressionExtractor::new(left_target_schema.as_ref(), alias_generator);
+ let mut right_extractor =
+ LeafExpressionExtractor::new(right_target_schema.as_ref(), alias_generator);
+
+ // Build column checker to route expressions to correct side
+ let mut column_checker = ColumnChecker::new(left_schema.as_ref(), right_schema.as_ref());
+
+ // Extract from `on` expressions (equijoin keys)
+ let mut new_on = Vec::with_capacity(join.on.len());
+ let mut any_extracted = false;
+
+ for (left_key, right_key) in &join.on {
+ // Left key should reference only left columns
+ let new_left = left_extractor.extract(left_key.clone())?;
+ if new_left.transformed {
+ any_extracted = true;
+ }
+
+ // Right key should reference only right columns
+ let new_right = right_extractor.extract(right_key.clone())?;
+ if new_right.transformed {
+ any_extracted = true;
+ }
+
+ new_on.push((new_left.data, new_right.data));
+ }
+
+ // Extract from `filter` expression
+ let new_filter = if let Some(ref filter) = join.filter {
+ let extracted = extract_from_join_filter(
+ filter.clone(),
+ &mut column_checker,
+ &mut left_extractor,
+ &mut right_extractor,
+ )?;
+ if extracted.transformed {
+ any_extracted = true;
+ }
+ Some(extracted.data)
+ } else {
+ None
+ };
+
+ if !any_extracted {
+ return Ok(Transformed::no(LogicalPlan::Join(join)));
+ }
+
+ // Save original schema before modifying inputs
+ let original_schema = Arc::clone(&join.schema);
+
+ // Build left extraction projection if needed
+ let new_left = if left_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&left_target) {
+ merge_into_extracted_projection(existing_proj, &left_extractor)?
+ } else {
+ left_extractor.build_projection_with_all_columns(left_target)?
+ };
+ Arc::new(rebuild_path(left_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.left)
+ };
+
+ // Build right extraction projection if needed
+ let new_right = if right_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&right_target) {
+ merge_into_extracted_projection(existing_proj, &right_extractor)?
+ } else {
+ right_extractor.build_projection_with_all_columns(right_target)?
+ };
+ Arc::new(rebuild_path(right_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.right)
+ };
+
+ // Create new Join with updated inputs and expressions
+ let new_join = datafusion_expr::logical_plan::Join::try_new(
+ new_left,
+ new_right,
+ new_on,
+ new_filter,
+ join.join_type,
+ join.join_constraint,
+ join.null_equality,
+ join.null_aware,
+ )?;
+
+ // Add recovery projection to restore original schema
+ // This hides the intermediate extracted expression columns
+ let recovered = build_recover_project_plan(original_schema.as_ref(), LogicalPlan::Join(new_join))?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a join filter expression.
+///
+/// For each sub-expression, determines if it references only left, only right,
+/// or both columns, and routes extractions accordingly.
+fn extract_from_join_filter(
+ filter: Expr,
+ column_checker: &mut ColumnChecker,
+ left_extractor: &mut LeafExpressionExtractor,
+ right_extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ filter.transform_down(|expr| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: expr,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
+
+ match expr.placement() {
+ ExpressionPlacement::MoveTowardsLeafNodes => {
+ // Check which side this expression belongs to
+ if column_checker.is_left_only(&expr) {
+ // Extract to left side
+ let col_ref = left_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else if column_checker.is_right_only(&expr) {
+ // Extract to right side
+ let col_ref = right_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else {
+ // References both sides - cannot extract, keep in place
+ // This shouldn't typically happen for MoveTowardsLeafNodes expressions
+ // but we handle it gracefully
+ Ok(Transformed::no(expr))
+ }
+ }
+ ExpressionPlacement::Column => {
+ // Track columns for pass-through on appropriate side
+ if let Expr::Column(col) = &expr {
+ if column_checker.is_left_only(&expr) {
+ left_extractor.columns_needed.insert(col.clone());
+ } else if column_checker.is_right_only(&expr) {
+ right_extractor.columns_needed.insert(col.clone());
+ }
+ }
+ Ok(Transformed::no(expr))
+ }
+ _ => {
+ // Continue recursing into children
+ Ok(Transformed::no(expr))
+ }
+ }
+ })
+}
+
+/// Evaluates the columns referenced in the given expression to see if they refer
+/// only to the left or right columns of a join.
+struct ColumnChecker<'a> {
+ left_schema: &'a DFSchema,
+ left_columns: Option<std::collections::HashSet<Column>>,
+ right_schema: &'a DFSchema,
+ right_columns: Option<std::collections::HashSet<Column>>,
+}
+
+impl<'a> ColumnChecker<'a> {
+ fn new(left_schema: &'a DFSchema, right_schema: &'a DFSchema) -> Self {
+ Self {
+ left_schema,
+ left_columns: None,
+ right_schema,
+ right_columns: None,
+ }
+ }
+
+ /// Return true if the expression references only columns from the left side
+ fn is_left_only(&mut self, predicate: &Expr) -> bool {
+ if self.left_columns.is_none() {
+ self.left_columns = Some(schema_columns(self.left_schema));
+ }
+ has_all_column_refs(predicate, self.left_columns.as_ref().unwrap())
+ }
+
+ /// Return true if the expression references only columns from the right side
+ fn is_right_only(&mut self, predicate: &Expr) -> bool {
+ if self.right_columns.is_none() {
+ self.right_columns = Some(schema_columns(self.right_schema));
+ }
+ has_all_column_refs(predicate, self.right_columns.as_ref().unwrap())
+ }
+}
+
+/// Returns all columns in the schema (both qualified and unqualified forms)
+fn schema_columns(schema: &DFSchema) -> std::collections::HashSet<Column> {
+ schema
+ .iter()
+ .flat_map(|(qualifier, field)| {
+ [
+ Column::new(qualifier.cloned(), field.name()),
+ Column::new_unqualified(field.name()),
+ ]
+ })
+ .collect()
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Aggregate nodes.
+///
+/// For Aggregates, we extract from:
+/// - Group-by expressions (full expressions or sub-expressions)
+/// - Arguments inside aggregate functions (NOT the aggregate function itself)
+///
+/// Uses CSE's two-level pattern with NamePreserver for stable name handling.
+fn extract_from_aggregate(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Aggregate(agg) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_group_names: Vec<_> = agg
+ .group_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+ let saved_aggr_names: Vec<_> = agg
+ .aggr_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+
+ // Find where to place extractions
+ let (target, path) = find_extraction_target(&agg.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from group-by expressions
+ let mut new_group_by = Vec::with_capacity(agg.group_expr.len());
+ let mut has_extractions = false;
+
+ for expr in &agg.group_expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_group_by.push(transformed.data);
+ }
+
+ // Extract from aggregate function arguments (not the function itself)
+ let mut new_aggr = Vec::with_capacity(agg.aggr_expr.len());
+
+ for expr in &agg.aggr_expr {
+ let transformed = extract_from_aggregate_args(expr.clone(), &mut extractor)?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_aggr.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Aggregate(agg)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Restore names in group-by expressions using NamePreserver
+ let restored_group_expr: Vec<Expr> = new_group_by
+ .into_iter()
+ .zip(saved_group_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Restore names in aggregate expressions using NamePreserver
+ let restored_aggr_expr: Vec<Expr> = new_aggr
+ .into_iter()
+ .zip(saved_aggr_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Create new Aggregate with restored names
+ // (no outer projection needed if names are properly preserved)
+ let new_agg = datafusion_expr::logical_plan::Aggregate::try_new(
+ Arc::new(rebuilt_input),
+ restored_group_expr,
+ restored_aggr_expr,
+ )?;
+
+ Ok(Transformed::yes(LogicalPlan::Aggregate(new_agg)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Projection nodes.
+///
+/// Uses CSE's two-level pattern (outer + inner projections only):
+/// - Inner projection: extraction with ALL columns passed through
+/// - Outer projection: rewritten expressions with restored names
+///
+/// This avoids the unstable 3-level structure that gets broken by OptimizeProjections.
+fn extract_from_projection(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Projection(proj) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Skip if this projection is fully extracted (only column references)
+ if is_fully_extracted(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Skip if this is already an extracted expression projection.
+ // This prevents re-extraction on subsequent optimizer passes.
+ if is_extracted_expr_projection(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
Review Comment:
I feel we can probably do better than this. I don't love the idea of identifying an alraedy extracted projection based on every expression having an alias with a certain pattern. I think we should be able to ask "is there anything to extract?" and then "can we push this down?" instead i.e. do the work but make it idempotent (does not change the plan) if there's no change.
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing extracted projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild the path from target back up to our node's input
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
+ .chain(
+ transformed
+ .data
+ .inputs()
+ .iter()
+ .skip(1)
+ .map(|p| (*p).clone()),
+ )
+ .collect();
+
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Use CSE's pattern: add recovery projection to restore original schema
+ let recovered = build_recover_project_plan(input_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Join nodes.
+///
+/// For Joins, we extract from:
+/// - `on` expressions: pairs of (left_key, right_key) for equijoin
+/// - `filter` expression: non-equi join conditions
+///
+/// Each expression is routed to the appropriate side (left or right) based on
+/// which columns it references. Expressions referencing columns from both sides
+/// cannot have sub-expressions extracted (they must remain in the filter).
+fn extract_from_join(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Join(join) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ let left_schema = join.left.schema();
+ let right_schema = join.right.schema();
+
+ // Create extractors for left and right sides
+ // Find extraction targets for each side (look through schema-preserving nodes)
+ let (left_target, left_path) = find_extraction_target(&join.left);
+ let (right_target, right_path) = find_extraction_target(&join.right);
+
+ let left_target_schema = Arc::clone(left_target.schema());
+ let right_target_schema = Arc::clone(right_target.schema());
+
+ let mut left_extractor =
+ LeafExpressionExtractor::new(left_target_schema.as_ref(), alias_generator);
+ let mut right_extractor =
+ LeafExpressionExtractor::new(right_target_schema.as_ref(), alias_generator);
+
+ // Build column checker to route expressions to correct side
+ let mut column_checker = ColumnChecker::new(left_schema.as_ref(), right_schema.as_ref());
+
+ // Extract from `on` expressions (equijoin keys)
+ let mut new_on = Vec::with_capacity(join.on.len());
+ let mut any_extracted = false;
+
+ for (left_key, right_key) in &join.on {
+ // Left key should reference only left columns
+ let new_left = left_extractor.extract(left_key.clone())?;
+ if new_left.transformed {
+ any_extracted = true;
+ }
+
+ // Right key should reference only right columns
+ let new_right = right_extractor.extract(right_key.clone())?;
+ if new_right.transformed {
+ any_extracted = true;
+ }
+
+ new_on.push((new_left.data, new_right.data));
+ }
+
+ // Extract from `filter` expression
+ let new_filter = if let Some(ref filter) = join.filter {
+ let extracted = extract_from_join_filter(
+ filter.clone(),
+ &mut column_checker,
+ &mut left_extractor,
+ &mut right_extractor,
+ )?;
+ if extracted.transformed {
+ any_extracted = true;
+ }
+ Some(extracted.data)
+ } else {
+ None
+ };
+
+ if !any_extracted {
+ return Ok(Transformed::no(LogicalPlan::Join(join)));
+ }
+
+ // Save original schema before modifying inputs
+ let original_schema = Arc::clone(&join.schema);
+
+ // Build left extraction projection if needed
+ let new_left = if left_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&left_target) {
+ merge_into_extracted_projection(existing_proj, &left_extractor)?
+ } else {
+ left_extractor.build_projection_with_all_columns(left_target)?
+ };
+ Arc::new(rebuild_path(left_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.left)
+ };
+
+ // Build right extraction projection if needed
+ let new_right = if right_extractor.has_extractions() {
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&right_target) {
+ merge_into_extracted_projection(existing_proj, &right_extractor)?
+ } else {
+ right_extractor.build_projection_with_all_columns(right_target)?
+ };
+ Arc::new(rebuild_path(right_path, LogicalPlan::Projection(extraction_proj))?)
+ } else {
+ Arc::clone(&join.right)
+ };
+
+ // Create new Join with updated inputs and expressions
+ let new_join = datafusion_expr::logical_plan::Join::try_new(
+ new_left,
+ new_right,
+ new_on,
+ new_filter,
+ join.join_type,
+ join.join_constraint,
+ join.null_equality,
+ join.null_aware,
+ )?;
+
+ // Add recovery projection to restore original schema
+ // This hides the intermediate extracted expression columns
+ let recovered = build_recover_project_plan(original_schema.as_ref(), LogicalPlan::Join(new_join))?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a join filter expression.
+///
+/// For each sub-expression, determines if it references only left, only right,
+/// or both columns, and routes extractions accordingly.
+fn extract_from_join_filter(
+ filter: Expr,
+ column_checker: &mut ColumnChecker,
+ left_extractor: &mut LeafExpressionExtractor,
+ right_extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ filter.transform_down(|expr| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: expr,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
+
+ match expr.placement() {
+ ExpressionPlacement::MoveTowardsLeafNodes => {
+ // Check which side this expression belongs to
+ if column_checker.is_left_only(&expr) {
+ // Extract to left side
+ let col_ref = left_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else if column_checker.is_right_only(&expr) {
+ // Extract to right side
+ let col_ref = right_extractor.add_extracted(expr)?;
+ Ok(Transformed::yes(col_ref))
+ } else {
+ // References both sides - cannot extract, keep in place
+ // This shouldn't typically happen for MoveTowardsLeafNodes expressions
+ // but we handle it gracefully
+ Ok(Transformed::no(expr))
+ }
+ }
+ ExpressionPlacement::Column => {
+ // Track columns for pass-through on appropriate side
+ if let Expr::Column(col) = &expr {
+ if column_checker.is_left_only(&expr) {
+ left_extractor.columns_needed.insert(col.clone());
+ } else if column_checker.is_right_only(&expr) {
+ right_extractor.columns_needed.insert(col.clone());
+ }
+ }
+ Ok(Transformed::no(expr))
+ }
+ _ => {
+ // Continue recursing into children
+ Ok(Transformed::no(expr))
+ }
+ }
+ })
+}
+
+/// Evaluates the columns referenced in the given expression to see if they refer
+/// only to the left or right columns of a join.
+struct ColumnChecker<'a> {
+ left_schema: &'a DFSchema,
+ left_columns: Option<std::collections::HashSet<Column>>,
+ right_schema: &'a DFSchema,
+ right_columns: Option<std::collections::HashSet<Column>>,
+}
+
+impl<'a> ColumnChecker<'a> {
+ fn new(left_schema: &'a DFSchema, right_schema: &'a DFSchema) -> Self {
+ Self {
+ left_schema,
+ left_columns: None,
+ right_schema,
+ right_columns: None,
+ }
+ }
+
+ /// Return true if the expression references only columns from the left side
+ fn is_left_only(&mut self, predicate: &Expr) -> bool {
+ if self.left_columns.is_none() {
+ self.left_columns = Some(schema_columns(self.left_schema));
+ }
+ has_all_column_refs(predicate, self.left_columns.as_ref().unwrap())
+ }
+
+ /// Return true if the expression references only columns from the right side
+ fn is_right_only(&mut self, predicate: &Expr) -> bool {
+ if self.right_columns.is_none() {
+ self.right_columns = Some(schema_columns(self.right_schema));
+ }
+ has_all_column_refs(predicate, self.right_columns.as_ref().unwrap())
+ }
+}
+
+/// Returns all columns in the schema (both qualified and unqualified forms)
+fn schema_columns(schema: &DFSchema) -> std::collections::HashSet<Column> {
+ schema
+ .iter()
+ .flat_map(|(qualifier, field)| {
+ [
+ Column::new(qualifier.cloned(), field.name()),
+ Column::new_unqualified(field.name()),
+ ]
+ })
+ .collect()
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Aggregate nodes.
+///
+/// For Aggregates, we extract from:
+/// - Group-by expressions (full expressions or sub-expressions)
+/// - Arguments inside aggregate functions (NOT the aggregate function itself)
+///
+/// Uses CSE's two-level pattern with NamePreserver for stable name handling.
+fn extract_from_aggregate(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Aggregate(agg) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_group_names: Vec<_> = agg
+ .group_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+ let saved_aggr_names: Vec<_> = agg
+ .aggr_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+
+ // Find where to place extractions
+ let (target, path) = find_extraction_target(&agg.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from group-by expressions
+ let mut new_group_by = Vec::with_capacity(agg.group_expr.len());
+ let mut has_extractions = false;
+
+ for expr in &agg.group_expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_group_by.push(transformed.data);
+ }
+
+ // Extract from aggregate function arguments (not the function itself)
+ let mut new_aggr = Vec::with_capacity(agg.aggr_expr.len());
+
+ for expr in &agg.aggr_expr {
+ let transformed = extract_from_aggregate_args(expr.clone(), &mut extractor)?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_aggr.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Aggregate(agg)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Restore names in group-by expressions using NamePreserver
+ let restored_group_expr: Vec<Expr> = new_group_by
+ .into_iter()
+ .zip(saved_group_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Restore names in aggregate expressions using NamePreserver
+ let restored_aggr_expr: Vec<Expr> = new_aggr
+ .into_iter()
+ .zip(saved_aggr_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Create new Aggregate with restored names
+ // (no outer projection needed if names are properly preserved)
+ let new_agg = datafusion_expr::logical_plan::Aggregate::try_new(
+ Arc::new(rebuilt_input),
+ restored_group_expr,
+ restored_aggr_expr,
+ )?;
+
+ Ok(Transformed::yes(LogicalPlan::Aggregate(new_agg)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Projection nodes.
+///
+/// Uses CSE's two-level pattern (outer + inner projections only):
+/// - Inner projection: extraction with ALL columns passed through
+/// - Outer projection: rewritten expressions with restored names
+///
+/// This avoids the unstable 3-level structure that gets broken by OptimizeProjections.
+fn extract_from_projection(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Projection(proj) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Skip if this projection is fully extracted (only column references)
+ if is_fully_extracted(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Skip if this is already an extracted expression projection.
+ // This prevents re-extraction on subsequent optimizer passes.
+ if is_extracted_expr_projection(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_names: Vec<_> = proj.expr.iter().map(|e| name_preserver.save(e)).collect();
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let (target, path) = find_extraction_target(&proj.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from projection expressions
+ let mut new_exprs = Vec::with_capacity(proj.expr.len());
+ let mut has_extractions = false;
+
+ for expr in &proj.expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_exprs.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_extracted_projection(&target) {
+ merge_into_extracted_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create outer projection with rewritten exprs + restored names
+ let final_exprs: Vec<Expr> = new_exprs
+ .into_iter()
+ .zip(saved_names)
+ .map(|(expr, saved_name)| saved_name.restore(expr))
+ .collect();
+
+ let outer_projection = Projection::try_new(final_exprs, Arc::new(rebuilt_input))?;
+
+ Ok(Transformed::yes(LogicalPlan::Projection(outer_projection)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from aggregate function arguments.
+///
+/// This extracts from inside the aggregate (e.g., from `sum(get_field(x, 'y'))`
+/// we extract `get_field(x, 'y')`), but NOT the aggregate function itself.
+fn extract_from_aggregate_args(
+ expr: Expr,
+ extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ match expr {
+ Expr::AggregateFunction(mut agg_func) => {
+ // Extract from arguments, not the function itself
+ let mut any_changed = false;
+ let mut new_args = Vec::with_capacity(agg_func.params.args.len());
+
+ for arg in agg_func.params.args {
+ let transformed = extractor.extract(arg)?;
+ if transformed.transformed {
+ any_changed = true;
+ }
+ new_args.push(transformed.data);
+ }
+
+ if any_changed {
+ agg_func.params.args = new_args;
+ Ok(Transformed::yes(Expr::AggregateFunction(agg_func)))
+ } else {
+ agg_func.params.args = new_args;
+ Ok(Transformed::no(Expr::AggregateFunction(agg_func)))
+ }
+ }
+ // For aliased aggregates, process the inner expression
+ Expr::Alias(alias) => {
+ let transformed = extract_from_aggregate_args(*alias.expr, extractor)?;
+ Ok(
+ transformed
+ .update_data(|e| e.alias_qualified(alias.relation, alias.name)),
+ )
+ }
+ // For other expressions, use regular extraction
+ other => extractor.extract(other),
+ }
+}
+
+// =============================================================================
+// Helper Functions for BottomUp Traversal
+// =============================================================================
+
+/// Traverses down through schema-preserving nodes to find where to place extractions.
+///
+/// Returns (target_node, path_to_rebuild) where:
+/// - target_node: the node above which to create extraction projection
+/// - path_to_rebuild: nodes between our input and target that must be rebuilt
+///
+/// Schema-preserving nodes that we can look through:
+/// - Filter, Sort, Limit: pass all input columns through unchanged
+/// - Passthrough projections: only column references
+///
+/// Barrier nodes where we stop:
+/// - TableScan, Join, Aggregate: these are extraction targets
+/// - Existing extracted expression projections: we merge into these
+/// - Any other node type
+fn find_extraction_target(
+ input: &Arc<LogicalPlan>,
+) -> (Arc<LogicalPlan>, Vec<LogicalPlan>) {
+ let mut current = Arc::clone(input);
+ let mut path = vec![];
+
+ loop {
+ match current.as_ref() {
+ // Look through schema-preserving nodes
+ LogicalPlan::Filter(f) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&f.input);
+ }
+ LogicalPlan::Sort(s) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&s.input);
+ }
+ LogicalPlan::Limit(l) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&l.input);
+ }
+ // Look through passthrough projections (only column references)
+ LogicalPlan::Projection(p) if is_passthrough_projection(p) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&p.input);
+ }
+ // Found existing extracted expression projection - will merge into it
+ LogicalPlan::Projection(p) if is_extracted_expr_projection(p) => {
+ return (current, path);
+ }
+ // Hit a barrier node - create new projection here
+ _ => {
+ return (current, path);
+ }
+ }
+ }
+}
+
+/// Returns true if the projection is a passthrough (only column references).
+fn is_passthrough_projection(proj: &Projection) -> bool {
+ proj.expr.iter().all(|e| matches!(e, Expr::Column(_)))
+}
+
+/// Returns true if the projection only has column references (nothing to extract).
+fn is_fully_extracted(proj: &Projection) -> bool {
+ proj.expr.iter().all(|e| {
+ matches!(e, Expr::Column(_))
+ || matches!(e, Expr::Alias(a) if matches!(a.expr.as_ref(), Expr::Column(_)))
+ })
+}
+
+/// If the target is an extracted expression projection, return it for merging.
+fn get_extracted_projection(target: &Arc<LogicalPlan>) -> Option<&Projection> {
+ if let LogicalPlan::Projection(p) = target.as_ref()
+ && is_extracted_expr_projection(p)
+ {
+ return Some(p);
+ }
+ None
+}
+
+/// Merges new extractions into an existing extracted expression projection.
+fn merge_into_extracted_projection(
+ existing: &Projection,
+ extractor: &LeafExpressionExtractor,
+) -> Result<Projection> {
+ let mut proj_exprs = existing.expr.clone();
+
+ // Build a map of existing expressions (by schema_name) to their aliases
+ let existing_extractions: IndexMap<String, String> = existing
+ .expr
+ .iter()
+ .filter_map(|e| {
+ if let Expr::Alias(alias) = e
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ let schema_name = alias.expr.schema_name().to_string();
+ return Some((schema_name, alias.name.clone()));
+ }
+ None
+ })
+ .collect();
+
+ // Add new extracted expressions, but only if not already present
+ for (schema_name, (expr, alias)) in &extractor.extracted {
+ if !existing_extractions.contains_key(schema_name) {
+ proj_exprs.push(expr.clone().alias(alias));
+ }
+ }
+
+ // Add any new pass-through columns that aren't already in the projection
+ let existing_cols: IndexSet<Column> = existing
+ .expr
+ .iter()
+ .filter_map(|e| {
+ if let Expr::Column(c) = e {
+ Some(c.clone())
+ } else {
+ None
+ }
+ })
+ .collect();
+
+ for col in &extractor.columns_needed {
+ if !existing_cols.contains(col) && extractor.input_schema.has_column(col) {
+ proj_exprs.push(Expr::Column(col.clone()));
+ }
+ }
+
+ Projection::try_new(proj_exprs, Arc::clone(&existing.input))
+}
+
+/// Rebuilds the path from extraction projection back up to original input.
+///
+/// Takes a list of nodes (in top-to-bottom order from input towards target)
+/// and rebuilds them with the new bottom input.
+///
+/// For passthrough projections, we update them to include ALL columns from
+/// the new input (including any new extracted expression columns that were merged).
+fn rebuild_path(path: Vec<LogicalPlan>, new_bottom: LogicalPlan) -> Result<LogicalPlan> {
+ let mut current = new_bottom;
+
+ // Rebuild path from bottom to top (reverse order)
+ for node in path.into_iter().rev() {
+ current = match node {
+ LogicalPlan::Filter(f) => {
+ LogicalPlan::Filter(Filter::try_new(f.predicate, Arc::new(current))?)
+ }
+ LogicalPlan::Sort(s) => LogicalPlan::Sort(Sort {
+ expr: s.expr,
+ input: Arc::new(current),
+ fetch: s.fetch,
+ }),
+ LogicalPlan::Limit(l) => LogicalPlan::Limit(Limit {
+ skip: l.skip,
+ fetch: l.fetch,
+ input: Arc::new(current),
+ }),
+ LogicalPlan::Projection(p) if is_passthrough_projection(&p) => {
Review Comment:
We should be able to handle non passthrough projections
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843396203
> BTW codex found a test that shows a single projection being extracted doesn't get pushed down
>
> I can make this a separate PR if you like
>
> note how the get_field is not pushed into the datasource:
>
> ```sql
> ###
> # Test 2.1b: Projection-only get_field (potential optimization target)
> ###
>
> query TT
> EXPLAIN SELECT s['label'] FROM simple_struct;
> ----
> logical_plan
> 01)Projection: get_field(simple_struct.s, Utf8("label"))
> 02)--TableScan: simple_struct projection=[s]
> physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, label) as simple_struct.s[label]], file_type=parquet
>
> # Verify correctness
> query T
> SELECT s['label'] FROM simple_struct ORDER BY s['label'];
> ----
> alpha
> beta
> delta
> epsilon
> gamma
> ```
Thanks for reporting. I’ll make a new PR with this test + join tests + benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843681761
> @alamb are there any of these that use structs? It seems like this has no impact on the benchmarks (good!) but maybe we should add some that hit the full rewrite?
I was more trying to quantify the effect on planning time of adding a new optimizer pass -- it seems like it is small but noticable slowdown (1-3%). I'll see if I can reproduce those results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843321158
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group get-field-pushdown-try-3 main
----- ------------------------ ----
logical_aggregate_with_join 1.01 643.3±10.62µs ? ?/sec 1.00 639.4±6.12µs ? ?/sec
logical_select_all_from_1000 1.00 10.3±0.09ms ? ?/sec 1.12 11.5±0.15ms ? ?/sec
logical_select_one_from_700 1.01 421.7±3.89µs ? ?/sec 1.00 415.8±2.36µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 380.8±12.39µs ? ?/sec 1.00 376.5±3.19µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.01 365.3±4.47µs ? ?/sec 1.00 361.5±11.63µs ? ?/sec
physical_intersection 1.00 1627.7±31.01µs ? ?/sec 1.00 1622.2±125.58µs ? ?/sec
physical_join_consider_sort 1.02 2.3±0.05ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_join_distinct 1.01 356.9±8.17µs ? ?/sec 1.00 353.0±3.93µs ? ?/sec
physical_many_self_joins 1.01 12.7±0.08ms ? ?/sec 1.00 12.5±0.26ms ? ?/sec
physical_plan_clickbench_all 1.01 203.2±1.51ms ? ?/sec 1.00 200.9±1.71ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.2±0.02ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.05 3.8±0.14ms ? ?/sec 1.00 3.7±0.10ms ? ?/sec
physical_plan_clickbench_q11 1.04 4.3±0.09ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q12 1.05 4.5±0.09ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q13 1.06 4.0±0.06ms ? ?/sec 1.00 3.8±0.12ms ? ?/sec
physical_plan_clickbench_q14 1.05 4.3±0.09ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q15 1.04 4.0±0.07ms ? ?/sec 1.00 3.9±0.09ms ? ?/sec
physical_plan_clickbench_q16 1.05 3.9±0.07ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.05 4.0±0.07ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.02 2.7±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.02 4.3±0.12ms ? ?/sec 1.00 4.2±0.09ms ? ?/sec
physical_plan_clickbench_q2 1.03 2.9±0.03ms ? ?/sec 1.00 2.8±0.06ms ? ?/sec
physical_plan_clickbench_q20 1.01 2.2±0.03ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.01 2.8±0.04ms ? ?/sec 1.00 2.8±0.05ms ? ?/sec
physical_plan_clickbench_q22 1.04 4.1±0.07ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q24 1.02 4.9±0.07ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.5±0.03ms ? ?/sec 1.00 3.5±0.09ms ? ?/sec
physical_plan_clickbench_q26 1.02 3.0±0.06ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q27 1.01 3.6±0.10ms ? ?/sec 1.00 3.5±0.06ms ? ?/sec
physical_plan_clickbench_q28 1.02 4.5±0.10ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q29 1.02 4.8±0.07ms ? ?/sec 1.00 4.7±0.04ms ? ?/sec
physical_plan_clickbench_q3 1.02 2.6±0.03ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.04 16.1±0.21ms ? ?/sec 1.00 15.5±0.22ms ? ?/sec
physical_plan_clickbench_q31 1.02 4.5±0.03ms ? ?/sec 1.00 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q32 1.02 4.5±0.04ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
physical_plan_clickbench_q33 1.02 3.7±0.09ms ? ?/sec 1.00 3.6±0.08ms ? ?/sec
physical_plan_clickbench_q34 1.02 3.3±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.01 3.4±0.03ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q36 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q37 1.02 4.7±0.09ms ? ?/sec 1.00 4.7±0.06ms ? ?/sec
physical_plan_clickbench_q38 1.01 4.7±0.04ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
physical_plan_clickbench_q39 1.01 4.1±0.07ms ? ?/sec 1.00 4.1±0.05ms ? ?/sec
physical_plan_clickbench_q4 1.03 2.2±0.04ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.01 5.0±0.05ms ? ?/sec 1.00 4.9±0.07ms ? ?/sec
physical_plan_clickbench_q41 1.01 4.3±0.03ms ? ?/sec 1.00 4.3±0.08ms ? ?/sec
physical_plan_clickbench_q42 1.01 4.3±0.07ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q43 1.01 4.6±0.06ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.3±0.08ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.01 2.3±0.02ms ? ?/sec 1.00 2.3±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.01 3.2±0.08ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q47 1.02 4.8±0.05ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
physical_plan_clickbench_q48 1.00 5.2±0.06ms ? ?/sec 1.00 5.3±0.19ms ? ?/sec
physical_plan_clickbench_q49 1.01 5.5±0.11ms ? ?/sec 1.00 5.4±0.08ms ? ?/sec
physical_plan_clickbench_q5 1.04 2.6±0.10ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.2±0.11ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q51 1.01 3.6±0.08ms ? ?/sec 1.00 3.6±0.07ms ? ?/sec
physical_plan_clickbench_q6 1.05 2.6±0.07ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.04 2.2±0.04ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.05 3.6±0.10ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.04 3.8±0.09ms ? ?/sec 1.00 3.6±0.08ms ? ?/sec
physical_plan_tpcds_all 1.00 1968.6±20.06ms ? ?/sec 1.00 1966.2±18.94ms ? ?/sec
physical_plan_tpch_all 1.00 129.4±1.21ms ? ?/sec 1.00 129.9±1.06ms ? ?/sec
physical_plan_tpch_q1 1.03 3.1±0.03ms ? ?/sec 1.00 3.0±0.06ms ? ?/sec
physical_plan_tpch_q10 1.01 7.4±0.06ms ? ?/sec 1.00 7.3±0.10ms ? ?/sec
physical_plan_tpch_q11 1.01 8.7±0.15ms ? ?/sec 1.00 8.6±0.17ms ? ?/sec
physical_plan_tpch_q12 1.01 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_tpch_q13 1.01 3.1±0.06ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q14 1.02 3.2±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_tpch_q16 1.01 5.3±0.04ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
physical_plan_tpch_q17 1.03 5.8±0.04ms ? ?/sec 1.00 5.6±0.07ms ? ?/sec
physical_plan_tpch_q18 1.01 6.1±0.06ms ? ?/sec 1.00 6.0±0.10ms ? ?/sec
physical_plan_tpch_q19 1.03 5.3±0.03ms ? ?/sec 1.00 5.1±0.10ms ? ?/sec
physical_plan_tpch_q2 1.03 12.7±0.13ms ? ?/sec 1.00 12.4±0.17ms ? ?/sec
physical_plan_tpch_q20 1.00 8.2±0.06ms ? ?/sec 1.01 8.3±0.15ms ? ?/sec
physical_plan_tpch_q21 1.01 10.3±0.10ms ? ?/sec 1.00 10.2±0.14ms ? ?/sec
physical_plan_tpch_q22 1.01 6.6±0.10ms ? ?/sec 1.00 6.5±0.06ms ? ?/sec
physical_plan_tpch_q3 1.02 5.7±0.04ms ? ?/sec 1.00 5.6±0.04ms ? ?/sec
physical_plan_tpch_q4 1.01 3.1±0.05ms ? ?/sec 1.00 3.0±0.07ms ? ?/sec
physical_plan_tpch_q5 1.01 6.0±0.04ms ? ?/sec 1.00 6.0±0.10ms ? ?/sec
physical_plan_tpch_q6 1.01 1617.3±16.07µs ? ?/sec 1.00 1604.1±25.37µs ? ?/sec
physical_plan_tpch_q7 1.02 7.4±0.08ms ? ?/sec 1.00 7.2±0.12ms ? ?/sec
physical_plan_tpch_q8 1.01 9.5±0.07ms ? ?/sec 1.00 9.3±0.09ms ? ?/sec
physical_plan_tpch_q9 1.00 6.8±0.04ms ? ?/sec 1.00 6.8±0.10ms ? ?/sec
physical_select_aggregates_from_200 1.00 17.7±0.13ms ? ?/sec 1.00 17.7±0.11ms ? ?/sec
physical_select_all_from_1000 1.00 23.6±0.24ms ? ?/sec 1.05 24.7±0.17ms ? ?/sec
physical_select_one_from_700 1.02 1357.1±8.44µs ? ?/sec 1.00 1327.1±11.77µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.02 11.3±0.09ms ? ?/sec 1.00 11.1±0.11ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.02 30.8±0.37ms ? ?/sec 1.00 30.2±0.37ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.01 200.9±2.34ms ? ?/sec 1.00 199.8±4.52ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.04 1148.2±7.76ms ? ?/sec 1.00 1104.5±15.61ms ? ?/sec
physical_theta_join_consider_sort 1.07 2.8±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_unnest_to_join 1.05 3.2±0.04ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.02 1612.1±18.59µs ? ?/sec 1.00 1587.3±11.05µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.06ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 1095.9±10.86µs ? ?/sec 1.00 1085.1±20.62µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.02 1280.3±30.33µs ? ?/sec 1.00 1261.3±17.40µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.02 1347.7±7.20µs ? ?/sec 1.00 1323.7±12.37µs ? ?/sec
with_param_values_many_columns 1.00 577.1±5.70µs ? ?/sec 1.05 606.5±14.52µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893435763
run benchmark sql_planning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2801311700
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -559,7 +559,19 @@ fn merge_consecutive_projections(proj: Projection) -> Result<Transformed<Project
metadata,
}) => rewrite_expr(*expr, &prev_projection).map(|result| {
result.update_data(|expr| {
- Expr::Alias(Alias::new(expr, relation, name).with_metadata(metadata))
+ // After substitution, the inner expression may now have the
+ // same schema_name as the alias (e.g. when an extraction
+ // alias like `__extracted_1 AS f(x)` is resolved back to
+ // `f(x)`). Wrapping in a redundant self-alias causes a
+ // cosmetic `f(x) AS f(x)` due to Display vs schema_name
+ // formatting differences. Drop the alias when it matches.
+ if expr.schema_name().to_string() == name {
Review Comment:
Generally we try to preserve metadata where "sensible". We probably don't do it well everywhere we should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3842885684
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2760985953
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::TopDown)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With TopDown traversal, we process parent nodes first, allowing us to
+/// merge expressions through child projections.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
Review Comment:
I’ll give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843653807
> Did you consider incorporating this logic directly into the `OptimizeProjections`? It seems like this transformation is really just a mechanism to enable `OptimizeProjections` 🤔
`OptimizeProjections` merges projections but it does not push them down. There is no logical optimizer rule for pushing down projections. We could move that part of this rule (the pushdown) into `OptimizeProjections` or a new rule but I’d find it easier to work it all in one place for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3845074249
> > BTW codex found a test that shows a single projection being extracted doesn't get pushed down
> > I can make this a separate PR if you like
> > note how the get_field is not pushed into the datasource:
> > ```sql
> > ###
> > # Test 2.1b: Projection-only get_field (potential optimization target)
> > ###
> >
> > query TT
> > EXPLAIN SELECT s['label'] FROM simple_struct;
> > ----
> > logical_plan
> > 01)Projection: get_field(simple_struct.s, Utf8("label"))
> > 02)--TableScan: simple_struct projection=[s]
> > physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, label) as simple_struct.s[label]], file_type=parquet
> >
> > # Verify correctness
> > query T
> > SELECT s['label'] FROM simple_struct ORDER BY s['label'];
> > ----
> > alpha
> > beta
> > delta
> > epsilon
> > gamma
> > ```
>
> Thanks for reporting. I’ll make a new PR with this test + join tests + benchmarks.
> > BTW codex found a test that shows a single projection being extracted doesn't get pushed down
> > I can make this a separate PR if you like
> > note how the get_field is not pushed into the datasource:
> > ```sql
> > ###
> > # Test 2.1b: Projection-only get_field (potential optimization target)
> > ###
> >
> > query TT
> > EXPLAIN SELECT s['label'] FROM simple_struct;
> > ----
> > logical_plan
> > 01)Projection: get_field(simple_struct.s, Utf8("label"))
> > 02)--TableScan: simple_struct projection=[s]
> > physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, label) as simple_struct.s[label]], file_type=parquet
> >
> > # Verify correctness
> > query T
> > SELECT s['label'] FROM simple_struct ORDER BY s['label'];
> > ----
> > alpha
> > beta
> > delta
> > epsilon
> > gamma
> > ```
>
> Thanks for reporting. I’ll make a new PR with this test + join tests + benchmarks.
Done in https://github.com/apache/datafusion/pull/20143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893679459
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and get-field-pushdown-try-3
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ get-field-pushdown-try-3 ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2378.84 ms │ 2372.50 ms │ no change │
│ QQuery 1 │ 912.70 ms │ 888.45 ms │ no change │
│ QQuery 2 │ 1835.54 ms │ 1784.51 ms │ no change │
│ QQuery 3 │ 1041.39 ms │ 1041.81 ms │ no change │
│ QQuery 4 │ 2243.42 ms │ 2269.33 ms │ no change │
│ QQuery 5 │ 28576.49 ms │ 28623.95 ms │ no change │
│ QQuery 6 │ 3934.49 ms │ 3868.65 ms │ no change │
│ QQuery 7 │ 3109.35 ms │ 2750.87 ms │ +1.13x faster │
└──────────┴─────────────┴──────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 44032.22ms │
│ Total Time (get-field-pushdown-try-3) │ 43600.07ms │
│ Average Time (HEAD) │ 5504.03ms │
│ Average Time (get-field-pushdown-try-3) │ 5450.01ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 7 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ get-field-pushdown-try-3 ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.61 ms │ 2.95 ms │ 1.13x slower │
│ QQuery 1 │ 52.27 ms │ 51.38 ms │ no change │
│ QQuery 2 │ 134.19 ms │ 134.95 ms │ no change │
│ QQuery 3 │ 156.66 ms │ 156.12 ms │ no change │
│ QQuery 4 │ 1132.12 ms │ 1085.88 ms │ no change │
│ QQuery 5 │ 1374.48 ms │ 1304.93 ms │ +1.05x faster │
│ QQuery 6 │ 6.58 ms │ 6.73 ms │ no change │
│ QQuery 7 │ 58.27 ms │ 56.55 ms │ no change │
│ QQuery 8 │ 1469.01 ms │ 1439.84 ms │ no change │
│ QQuery 9 │ 1863.96 ms │ 1811.71 ms │ no change │
│ QQuery 10 │ 352.53 ms │ 352.26 ms │ no change │
│ QQuery 11 │ 403.81 ms │ 408.45 ms │ no change │
│ QQuery 12 │ 1295.56 ms │ 1237.51 ms │ no change │
│ QQuery 13 │ 1995.46 ms │ 1966.52 ms │ no change │
│ QQuery 14 │ 1322.11 ms │ 1248.40 ms │ +1.06x faster │
│ QQuery 15 │ 1303.35 ms │ 1252.02 ms │ no change │
│ QQuery 16 │ 2596.00 ms │ 2579.25 ms │ no change │
│ QQuery 17 │ 2550.55 ms │ 2571.64 ms │ no change │
│ QQuery 18 │ 5817.54 ms │ 4929.30 ms │ +1.18x faster │
│ QQuery 19 │ 123.43 ms │ 123.07 ms │ no change │
│ QQuery 20 │ 1985.26 ms │ 1906.36 ms │ no change │
│ QQuery 21 │ 2251.23 ms │ 2184.85 ms │ no change │
│ QQuery 22 │ 3829.70 ms │ 3755.92 ms │ no change │
│ QQuery 23 │ 22238.10 ms │ 12192.52 ms │ +1.82x faster │
│ QQuery 24 │ 230.58 ms │ 209.86 ms │ +1.10x faster │
│ QQuery 25 │ 486.07 ms │ 468.12 ms │ no change │
│ QQuery 26 │ 221.57 ms │ 210.33 ms │ +1.05x faster │
│ QQuery 27 │ 2765.23 ms │ 2676.26 ms │ no change │
│ QQuery 28 │ 22399.91 ms │ 23856.14 ms │ 1.07x slower │
│ QQuery 29 │ 980.52 ms │ 973.80 ms │ no change │
│ QQuery 30 │ 1352.40 ms │ 1296.38 ms │ no change │
│ QQuery 31 │ 1386.20 ms │ 1374.02 ms │ no change │
│ QQuery 32 │ 4893.99 ms │ 4884.58 ms │ no change │
│ QQuery 33 │ 5839.86 ms │ 5553.41 ms │ no change │
│ QQuery 34 │ 6277.24 ms │ 6112.95 ms │ no change │
│ QQuery 35 │ 1980.72 ms │ 1939.60 ms │ no change │
│ QQuery 36 │ 203.78 ms │ 194.15 ms │ no change │
│ QQuery 37 │ 78.11 ms │ 75.50 ms │ no change │
│ QQuery 38 │ 117.64 ms │ 114.63 ms │ no change │
│ QQuery 39 │ 363.57 ms │ 348.46 ms │ no change │
│ QQuery 40 │ 43.35 ms │ 39.39 ms │ +1.10x faster │
│ QQuery 41 │ 35.69 ms │ 35.81 ms │ no change │
│ QQuery 42 │ 31.93 ms │ 32.20 ms │ no change │
└───────────┴─────────────┴──────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 104003.15ms │
│ Total Time (get-field-pushdown-try-3) │ 93154.71ms │
│ Average Time (HEAD) │ 2418.68ms │
│ Average Time (get-field-pushdown-try-3) │ 2166.39ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 34 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ get-field-pushdown-try-3 ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 104.69 ms │ 105.07 ms │ no change │
│ QQuery 2 │ 33.19 ms │ 35.39 ms │ 1.07x slower │
│ QQuery 3 │ 40.88 ms │ 42.30 ms │ no change │
│ QQuery 4 │ 31.42 ms │ 31.77 ms │ no change │
│ QQuery 5 │ 93.36 ms │ 92.67 ms │ no change │
│ QQuery 6 │ 21.04 ms │ 21.29 ms │ no change │
│ QQuery 7 │ 164.42 ms │ 160.41 ms │ no change │
│ QQuery 8 │ 43.14 ms │ 43.82 ms │ no change │
│ QQuery 9 │ 101.73 ms │ 105.41 ms │ no change │
│ QQuery 10 │ 66.68 ms │ 68.82 ms │ no change │
│ QQuery 11 │ 19.90 ms │ 19.73 ms │ no change │
│ QQuery 12 │ 51.78 ms │ 53.37 ms │ no change │
│ QQuery 13 │ 51.09 ms │ 50.91 ms │ no change │
│ QQuery 14 │ 15.88 ms │ 15.66 ms │ no change │
│ QQuery 15 │ 31.33 ms │ 31.00 ms │ no change │
│ QQuery 16 │ 29.56 ms │ 29.30 ms │ no change │
│ QQuery 17 │ 147.13 ms │ 149.44 ms │ no change │
│ QQuery 18 │ 290.47 ms │ 289.92 ms │ no change │
│ QQuery 19 │ 39.74 ms │ 39.99 ms │ no change │
│ QQuery 20 │ 57.41 ms │ 58.65 ms │ no change │
│ QQuery 21 │ 197.35 ms │ 192.99 ms │ no change │
│ QQuery 22 │ 23.16 ms │ 23.15 ms │ no change │
└───────────┴───────────┴──────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1655.34ms │
│ Total Time (get-field-pushdown-try-3) │ 1661.08ms │
│ Average Time (HEAD) │ 75.24ms │
│ Average Time (get-field-pushdown-try-3) │ 75.50ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└─────────────────────────────────────────┴───────────┘
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893495003
🤖 `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing get-field-pushdown-try-3 (df4104b9e640600c00e2ebf26519557e2c161257) to f9d37cf769454fd1ce3f5af74ffd9323664a4c6d [diff](https://github.com/apache/datafusion/compare/f9d37cf769454fd1ce3f5af74ffd9323664a4c6d..df4104b9e640600c00e2ebf26519557e2c161257) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756632022
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -1339,7 +1359,247 @@ SELECT id, s['value'] FROM simple_struct ORDER BY id, s['value'];
5 250
#####################
-# Section 12: Cleanup
+# Section 12: Join Tests - get_field Extraction from Join Nodes
Review Comment:
I can break these out into another PR to reduce the diff if that's helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2767540138
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1295,6 +1297,18 @@ fn rewrite_projection(
predicates: Vec<Expr>,
mut projection: Projection,
) -> Result<(Transformed<LogicalPlan>, Option<Expr>)> {
+ // Note: This check coordinates with ExtractLeafExpressions optimizer rule.
+ // See extract_leaf_expressions.rs for details on why these projections exist.
+ // Don't push filters through extracted expression projections.
+ // Pushing filters through would rewrite expressions like `__datafusion_extracted_1 > 150`
+ // back to `get_field(s,'value') > 150`, undoing the extraction.
+ if is_extracted_expr_projection(&projection) {
Review Comment:
Won't projection pushdown push the projection later inside `Filter` again? What does a final plan look like?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893080561
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing get-field-pushdown-try-3 (df4104b9e640600c00e2ebf26519557e2c161257) to f9d37cf769454fd1ce3f5af74ffd9323664a4c6d [diff](https://github.com/apache/datafusion/compare/f9d37cf769454fd1ce3f5af74ffd9323664a4c6d..df4104b9e640600c00e2ebf26519557e2c161257)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=get-field-pushdown-try-3
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2800896258
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -954,27 +965,29 @@ EXPLAIN SELECT (id + s['value']) * (id + s['value']) as id_and_value FROM simple
----
logical_plan
01)Projection: __common_expr_1 * __common_expr_1 AS id_and_value
-02)--Projection: simple_struct.id + get_field(simple_struct.s, Utf8("value")) AS __common_expr_1
+02)--Projection: simple_struct.id + __datafusion_extracted_2 AS __common_expr_1
03)----Filter: simple_struct.id > Int64(2)
-04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
+04)------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_2, simple_struct.id
+05)--------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
physical_plan
01)ProjectionExec: expr=[__common_expr_1@0 * __common_expr_1@0 as id_and_value]
-02)--ProjectionExec: expr=[id@0 + get_field(s@1, value) as __common_expr_1]
-03)----FilterExec: id@0 > 2
-04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, s], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+02)--ProjectionExec: expr=[id@1 + __datafusion_extracted_2@0 as __common_expr_1]
+03)----FilterExec: id@1 > 2
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[get_field(s@1, value) as __datafusion_extracted_2, id], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
query TT
EXPLAIN SELECT s['value'] + s['value'] as doubled FROM simple_struct WHERE id > 2;
----
logical_plan
-01)Projection: get_field(simple_struct.s, Utf8("value")) + get_field(simple_struct.s, Utf8("value")) AS doubled
+01)Projection: __datafusion_extracted_1 + __datafusion_extracted_1 AS doubled
02)--Filter: simple_struct.id > Int64(2)
-03)----TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
+03)----Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_1, simple_struct.id
+04)------TableScan: simple_struct projection=[id, s], partial_filters=[simple_struct.id > Int64(2)]
physical_plan
-01)ProjectionExec: expr=[get_field(s@0, value) + get_field(s@0, value) as doubled]
-02)--FilterExec: id@0 > 2, projection=[s@1]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/projection_pushdown/simple.parquet]]}, projection=[id, s], file_type=parquet, predicate=id@0 > 2, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 > 2, required_guarantees=[]
+01)ProjectionExec: expr=[__datafusion_extracted_1@0 + __datafusion_extracted_1@0 as doubled]
Review Comment:
this is a nice plan -- it shows that the field is extracted once and then the expression is computed after the filter 👍
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -283,13 +1295,20 @@ mod tests {
TableScan: test projection=[id, user]
## After Extraction
- (same as original)
+ Projection: test.id
+ Projection: test.id, test.user
+ Filter: __datafusion_extracted_1 = Utf8("active")
+ Projection: leaf_udf(test.user, Utf8("status")) AS __datafusion_extracted_1, test.id, test.user
+ TableScan: test projection=[id, user]
## After Pushdown
(same as after extraction)
## Optimized
- (same as after pushdown)
+ Projection: test.id
+ Filter: __datafusion_extracted_1 = Utf8("active")
+ Projection: leaf_udf(test.user, Utf8("status")) AS __datafusion_extracted_1, test.id
Review Comment:
this shows the result of this rewrite pretty nicely -- the extraction was pushed below the filter (though in this case it probably doesn't make any difference to the performance)
##########
datafusion/sqllogictest/test_files/insert.slt:
##########
@@ -165,7 +165,7 @@ ORDER BY c1
----
logical_plan
01)Dml: op=[Insert Into] table=[table_without_values]
-02)--Projection: a1 AS a1, a2 AS a2
+02)--Projection: a1, a2
Review Comment:
that is certainly nicer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2801309294
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -95,9 +101,533 @@ impl OptimizerRule for ExtractLeafExpressions {
fn rewrite(
&self,
plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
+ config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
- Ok(Transformed::no(plan))
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// Works for any number of inputs (0, 1, 2, …N). For multi-input nodes
+/// like Join, each extracted sub-expression is routed to the correct input
+/// by checking which input's schema contains all of the expression's column
+/// references.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Only extract from plan types whose output schema is predictable after
+ // expression rewriting. Nodes like Window derive column names from
+ // their expressions, so rewriting `get_field` inside a window function
+ // changes the output schema and breaks the recovery projection.
+ if !matches!(
+ &plan,
+ LogicalPlan::Aggregate(_)
+ | LogicalPlan::Filter(_)
+ | LogicalPlan::Sort(_)
+ | LogicalPlan::Limit(_)
+ | LogicalPlan::Join(_)
+ ) {
+ return Ok(Transformed::no(plan));
+ }
+
+ let inputs = plan.inputs();
+ if inputs.is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ // Save original output schema before any transformation
+ let original_schema = Arc::clone(plan.schema());
+
+ // Build per-input schemas from borrowed inputs (before plan is consumed
+ // by map_expressions). We only need schemas and column sets for routing;
+ // the actual inputs are cloned later only if extraction succeeds.
+ let input_schemas: Vec<Arc<DFSchema>> =
+ inputs.iter().map(|i| Arc::clone(i.schema())).collect();
+
+ // Build per-input extractors
+ let mut extractors: Vec<LeafExpressionExtractor> = input_schemas
+ .iter()
+ .map(|schema| LeafExpressionExtractor::new(schema.as_ref(), alias_generator))
+ .collect();
+
+ // Build per-input column sets for routing expressions to the correct input
+ let input_column_sets: Vec<std::collections::HashSet<Column>> = input_schemas
+ .iter()
+ .map(|schema| schema_columns(schema.as_ref()))
+ .collect();
+
+ // Transform expressions via map_expressions with routing
+ let transformed = plan.map_expressions(|expr| {
+ routing_extract(expr, &mut extractors, &input_column_sets)
+ })?;
+
+ // If no expressions were rewritten, nothing was extracted
+ if !transformed.transformed {
+ return Ok(transformed);
+ }
+
+ // Clone inputs now that we know extraction succeeded. Wrap in Arc
+ // upfront since build_extraction_projection expects &Arc<LogicalPlan>.
+ let owned_inputs: Vec<Arc<LogicalPlan>> = transformed
+ .data
+ .inputs()
+ .into_iter()
+ .map(|i| Arc::new(i.clone()))
+ .collect();
+
+ // Build per-input extraction projections (None means no extractions for that input)
+ let new_inputs: Vec<LogicalPlan> = owned_inputs
+ .into_iter()
+ .zip(extractors.iter())
+ .map(|(input_arc, extractor)| {
+ match extractor.build_extraction_projection(&input_arc)? {
+ Some(plan) => Ok(plan),
+ // No extractions for this input — recover the LogicalPlan
+ // without cloning (refcount is 1 since build returned None).
+ None => {
+ Ok(Arc::try_unwrap(input_arc).unwrap_or_else(|arc| (*arc).clone()))
+ }
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // Rebuild the plan keeping its rewritten expressions but replacing
+ // inputs with the new extraction projections.
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Add recovery projection if the output schema changed
+ let recovered = build_recovery_projection(original_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Given an expression, returns the index of the input whose columns fully
+/// cover the expression's column references.
+/// Returns `None` if the expression references columns from multiple inputs
+/// or if multiple inputs match (ambiguous, e.g. unqualified columns present
+/// in both sides of a join).
+fn find_owning_input(
+ expr: &Expr,
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Option<usize> {
+ let mut found = None;
+ for (idx, cols) in input_column_sets.iter().enumerate() {
+ if has_all_column_refs(expr, cols) {
+ if found.is_some() {
+ // Ambiguous — multiple inputs match
+ return None;
+ }
+ found = Some(idx);
+ }
+ }
+ found
+}
+
+/// Walks an expression tree top-down, extracting `MoveTowardsLeafNodes`
+/// sub-expressions and routing each to the correct per-input extractor.
+fn routing_extract(
+ expr: Expr,
+ extractors: &mut [LeafExpressionExtractor],
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Result<Transformed<Expr>> {
+ expr.transform_down(|e| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &e
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: e,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
Review Comment:
I left a comment on the prefix constant. I don't know that there's a better way to document this at the moment. If there are issues we can work through them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893921792
🤖 `./gh_compare_branch_bench.sh` [compare_branch_bench.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/compare_branch_bench.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing get-field-pushdown-try-3 (40b6c5388e13045955101e5a5b5fb76aecfe5bb0) to af5f47046561b98df0e85d1183faea8f9b52066f [diff](https://github.com/apache/datafusion/compare/af5f47046561b98df0e85d1183faea8f9b52066f..40b6c5388e13045955101e5a5b5fb76aecfe5bb0)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=parquet --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=get-field-pushdown-try-3
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3894145918
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group get-field-pushdown-try-3 main
----- ------------------------ ----
logical_aggregate_with_join 1.00 634.9±4.85µs ? ?/sec 1.01 640.0±7.59µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 290.8±3.16µs ? ?/sec 1.01 294.0±4.99µs ? ?/sec
logical_select_all_from_1000 1.00 10.9±0.04ms ? ?/sec 1.04 11.3±0.08ms ? ?/sec
logical_select_one_from_700 1.00 412.4±3.29µs ? ?/sec 1.00 413.6±2.00µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 370.7±2.24µs ? ?/sec 1.01 373.4±4.39µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 356.0±2.14µs ? ?/sec 1.01 359.2±4.88µs ? ?/sec
physical_intersection 1.08 1722.8±42.12µs ? ?/sec 1.00 1600.3±20.78µs ? ?/sec
physical_join_consider_sort 1.12 2.5±0.02ms ? ?/sec 1.00 2.3±0.05ms ? ?/sec
physical_join_distinct 1.00 348.6±1.31µs ? ?/sec 1.01 351.1±8.63µs ? ?/sec
physical_many_self_joins 1.13 14.2±0.14ms ? ?/sec 1.00 12.6±0.08ms ? ?/sec
physical_plan_clickbench_all 1.03 212.7±2.62ms ? ?/sec 1.00 205.7±3.38ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.2±0.04ms ? ?/sec 1.00 2.2±0.04ms ? ?/sec
physical_plan_clickbench_q10 1.03 3.8±0.05ms ? ?/sec 1.00 3.7±0.10ms ? ?/sec
physical_plan_clickbench_q11 1.03 4.3±0.14ms ? ?/sec 1.00 4.2±0.08ms ? ?/sec
physical_plan_clickbench_q12 1.02 4.5±0.14ms ? ?/sec 1.00 4.4±0.10ms ? ?/sec
physical_plan_clickbench_q13 1.02 3.9±0.08ms ? ?/sec 1.00 3.8±0.07ms ? ?/sec
physical_plan_clickbench_q14 1.01 4.3±0.04ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q15 1.02 4.0±0.10ms ? ?/sec 1.00 4.0±0.11ms ? ?/sec
physical_plan_clickbench_q16 1.02 3.8±0.04ms ? ?/sec 1.00 3.7±0.06ms ? ?/sec
physical_plan_clickbench_q17 1.01 3.9±0.08ms ? ?/sec 1.00 3.9±0.07ms ? ?/sec
physical_plan_clickbench_q18 1.02 2.7±0.07ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.01 4.4±0.07ms ? ?/sec 1.00 4.3±0.22ms ? ?/sec
physical_plan_clickbench_q2 1.04 2.9±0.07ms ? ?/sec 1.00 2.8±0.05ms ? ?/sec
physical_plan_clickbench_q20 1.01 2.2±0.02ms ? ?/sec 1.00 2.2±0.07ms ? ?/sec
physical_plan_clickbench_q21 1.04 2.9±0.07ms ? ?/sec 1.00 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q22 1.02 4.1±0.03ms ? ?/sec 1.00 4.0±0.06ms ? ?/sec
physical_plan_clickbench_q23 1.04 4.4±0.03ms ? ?/sec 1.00 4.3±0.14ms ? ?/sec
physical_plan_clickbench_q24 1.07 5.3±0.13ms ? ?/sec 1.00 5.0±0.08ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.6±0.03ms ? ?/sec 1.00 3.5±0.06ms ? ?/sec
physical_plan_clickbench_q26 1.03 3.0±0.06ms ? ?/sec 1.00 2.9±0.09ms ? ?/sec
physical_plan_clickbench_q27 1.03 3.7±0.05ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q28 1.05 4.8±0.19ms ? ?/sec 1.00 4.6±0.08ms ? ?/sec
physical_plan_clickbench_q29 1.04 5.0±0.09ms ? ?/sec 1.00 4.8±0.09ms ? ?/sec
physical_plan_clickbench_q3 1.04 2.6±0.03ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q30 1.06 17.1±0.20ms ? ?/sec 1.00 16.1±0.25ms ? ?/sec
physical_plan_clickbench_q31 1.05 4.7±0.15ms ? ?/sec 1.00 4.5±0.13ms ? ?/sec
physical_plan_clickbench_q32 1.00 4.7±0.09ms ? ?/sec 1.00 4.7±0.20ms ? ?/sec
physical_plan_clickbench_q33 1.03 3.8±0.06ms ? ?/sec 1.00 3.7±0.06ms ? ?/sec
physical_plan_clickbench_q34 1.02 3.4±0.06ms ? ?/sec 1.00 3.3±0.07ms ? ?/sec
physical_plan_clickbench_q35 1.01 3.5±0.04ms ? ?/sec 1.00 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q36 1.04 4.5±0.10ms ? ?/sec 1.00 4.3±0.15ms ? ?/sec
physical_plan_clickbench_q37 1.06 5.0±0.14ms ? ?/sec 1.00 4.7±0.06ms ? ?/sec
physical_plan_clickbench_q38 1.04 5.0±0.11ms ? ?/sec 1.00 4.8±0.08ms ? ?/sec
physical_plan_clickbench_q39 1.04 4.3±0.13ms ? ?/sec 1.00 4.2±0.09ms ? ?/sec
physical_plan_clickbench_q4 1.02 2.2±0.03ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.05 5.2±0.16ms ? ?/sec 1.00 5.0±0.08ms ? ?/sec
physical_plan_clickbench_q41 1.04 4.5±0.12ms ? ?/sec 1.00 4.4±0.13ms ? ?/sec
physical_plan_clickbench_q42 1.02 4.4±0.08ms ? ?/sec 1.00 4.4±0.12ms ? ?/sec
physical_plan_clickbench_q43 1.02 4.8±0.12ms ? ?/sec 1.00 4.7±0.11ms ? ?/sec
physical_plan_clickbench_q44 1.03 2.4±0.03ms ? ?/sec 1.00 2.3±0.03ms ? ?/sec
physical_plan_clickbench_q45 1.03 2.4±0.04ms ? ?/sec 1.00 2.3±0.04ms ? ?/sec
physical_plan_clickbench_q46 1.03 3.4±0.04ms ? ?/sec 1.00 3.3±0.05ms ? ?/sec
physical_plan_clickbench_q47 1.06 5.1±0.11ms ? ?/sec 1.00 4.8±0.05ms ? ?/sec
physical_plan_clickbench_q48 1.03 5.5±0.17ms ? ?/sec 1.00 5.3±0.16ms ? ?/sec
physical_plan_clickbench_q49 1.02 5.7±0.13ms ? ?/sec 1.00 5.6±0.13ms ? ?/sec
physical_plan_clickbench_q5 1.02 2.6±0.04ms ? ?/sec 1.00 2.6±0.06ms ? ?/sec
physical_plan_clickbench_q50 1.07 4.5±0.10ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q51 1.05 3.7±0.11ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q6 1.03 2.6±0.03ms ? ?/sec 1.00 2.5±0.06ms ? ?/sec
physical_plan_clickbench_q7 1.03 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.03 3.6±0.09ms ? ?/sec 1.00 3.5±0.08ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.7±0.06ms ? ?/sec 1.00 3.7±0.12ms ? ?/sec
physical_plan_struct_join_agg_sort 1.25 3.4±0.03ms ? ?/sec 1.00 2.7±0.12ms ? ?/sec
physical_plan_tpcds_all 1.05 2.0±0.01s ? ?/sec 1.00 1940.6±19.97ms ? ?/sec
physical_plan_tpch_all 1.04 132.3±2.66ms ? ?/sec 1.00 127.5±2.56ms ? ?/sec
physical_plan_tpch_q1 1.07 3.2±0.09ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
physical_plan_tpch_q10 1.05 7.7±0.18ms ? ?/sec 1.00 7.3±0.28ms ? ?/sec
physical_plan_tpch_q11 1.04 8.9±0.10ms ? ?/sec 1.00 8.5±0.11ms ? ?/sec
physical_plan_tpch_q12 1.04 3.2±0.06ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_tpch_q13 1.02 3.1±0.05ms ? ?/sec 1.00 3.0±0.10ms ? ?/sec
physical_plan_tpch_q14 1.05 3.2±0.09ms ? ?/sec 1.00 3.0±0.05ms ? ?/sec
physical_plan_tpch_q16 1.04 5.4±0.10ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
physical_plan_tpch_q17 1.05 5.9±0.23ms ? ?/sec 1.00 5.6±0.15ms ? ?/sec
physical_plan_tpch_q18 1.03 6.2±0.11ms ? ?/sec 1.00 6.0±0.04ms ? ?/sec
physical_plan_tpch_q19 1.07 5.4±0.10ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
physical_plan_tpch_q2 1.03 12.9±0.33ms ? ?/sec 1.00 12.5±0.42ms ? ?/sec
physical_plan_tpch_q20 1.04 8.3±0.09ms ? ?/sec 1.00 8.0±0.08ms ? ?/sec
physical_plan_tpch_q21 1.04 10.5±0.11ms ? ?/sec 1.00 10.2±0.11ms ? ?/sec
physical_plan_tpch_q22 1.04 6.7±0.26ms ? ?/sec 1.00 6.4±0.05ms ? ?/sec
physical_plan_tpch_q3 1.06 5.9±0.15ms ? ?/sec 1.00 5.6±0.04ms ? ?/sec
physical_plan_tpch_q4 1.04 3.1±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_tpch_q5 1.04 6.2±0.08ms ? ?/sec 1.00 6.0±0.20ms ? ?/sec
physical_plan_tpch_q6 1.03 1660.7±35.91µs ? ?/sec 1.00 1604.9±49.97µs ? ?/sec
physical_plan_tpch_q7 1.05 7.4±0.07ms ? ?/sec 1.00 7.1±0.07ms ? ?/sec
physical_plan_tpch_q8 1.04 9.7±0.09ms ? ?/sec 1.00 9.3±0.08ms ? ?/sec
physical_plan_tpch_q9 1.05 6.9±0.04ms ? ?/sec 1.00 6.6±0.07ms ? ?/sec
physical_select_aggregates_from_200 1.33 23.5±0.11ms ? ?/sec 1.00 17.6±0.11ms ? ?/sec
physical_select_all_from_1000 1.01 25.0±0.18ms ? ?/sec 1.00 24.7±0.16ms ? ?/sec
physical_select_one_from_700 1.13 1496.9±22.01µs ? ?/sec 1.00 1319.1±17.32µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.03 11.4±0.18ms ? ?/sec 1.00 11.0±0.08ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.05 31.5±1.00ms ? ?/sec 1.00 30.1±0.99ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.03 203.4±2.26ms ? ?/sec 1.00 198.0±2.32ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.08 1176.4±8.87ms ? ?/sec 1.00 1088.4±10.67ms ? ?/sec
physical_theta_join_consider_sort 1.10 2.9±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_unnest_to_join 1.12 3.4±0.13ms ? ?/sec 1.00 3.1±0.07ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.02 1599.4±26.09µs ? ?/sec 1.00 1564.5±25.36µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.02 3.0±0.09ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.01 1109.4±29.27µs ? ?/sec 1.00 1094.7±38.72µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.02 1267.1±13.74µs ? ?/sec 1.00 1242.4±7.73µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.02 1344.2±26.21µs ? ?/sec 1.00 1315.5±15.96µs ? ?/sec
with_param_values_many_columns 1.00 590.1±4.09µs ? ?/sec 1.01 598.4±5.07µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2801307756
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -559,7 +559,19 @@ fn merge_consecutive_projections(proj: Projection) -> Result<Transformed<Project
metadata,
}) => rewrite_expr(*expr, &prev_projection).map(|result| {
result.update_data(|expr| {
- Expr::Alias(Alias::new(expr, relation, name).with_metadata(metadata))
+ // After substitution, the inner expression may now have the
+ // same schema_name as the alias (e.g. when an extraction
+ // alias like `__extracted_1 AS f(x)` is resolved back to
+ // `f(x)`). Wrapping in a redundant self-alias causes a
+ // cosmetic `f(x) AS f(x)` due to Display vs schema_name
+ // formatting differences. Drop the alias when it matches.
+ if expr.schema_name().to_string() == name {
+ expr
+ } else {
+ Expr::Alias(
+ Alias::new(expr, relation, name).with_metadata(metadata),
+ )
+ }
Review Comment:
Good suggestion. I incorporated this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #20117:
URL: https://github.com/apache/datafusion/pull/20117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893079976
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "ethan-tyler (via GitHub)" <gi...@apache.org>.
ethan-tyler commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2800731660
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -559,7 +559,19 @@ fn merge_consecutive_projections(proj: Projection) -> Result<Transformed<Project
metadata,
}) => rewrite_expr(*expr, &prev_projection).map(|result| {
result.update_data(|expr| {
- Expr::Alias(Alias::new(expr, relation, name).with_metadata(metadata))
+ // After substitution, the inner expression may now have the
+ // same schema_name as the alias (e.g. when an extraction
+ // alias like `__extracted_1 AS f(x)` is resolved back to
+ // `f(x)`). Wrapping in a redundant self-alias causes a
+ // cosmetic `f(x) AS f(x)` due to Display vs schema_name
+ // formatting differences. Drop the alias when it matches.
+ if expr.schema_name().to_string() == name {
+ expr
+ } else {
+ Expr::Alias(
+ Alias::new(expr, relation, name).with_metadata(metadata),
+ )
+ }
Review Comment:
self alias stripping drops `Alias.metadata` and `SavedName::restore` doesn't recover it. Given self aliases after rewrite will be more common, I think it's best to guard on `metadata.is_none()` before stripping.
Something like:
```suggestion
if metadata.is_none() && expr.schema_name().to_string() == name {
expr
} else {
Expr::Alias(
Alias::new(expr, relation, name)
.with_metadata(metadata),
)
}
```
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -559,7 +559,19 @@ fn merge_consecutive_projections(proj: Projection) -> Result<Transformed<Project
metadata,
}) => rewrite_expr(*expr, &prev_projection).map(|result| {
result.update_data(|expr| {
- Expr::Alias(Alias::new(expr, relation, name).with_metadata(metadata))
+ // After substitution, the inner expression may now have the
+ // same schema_name as the alias (e.g. when an extraction
+ // alias like `__extracted_1 AS f(x)` is resolved back to
+ // `f(x)`). Wrapping in a redundant self-alias causes a
+ // cosmetic `f(x) AS f(x)` due to Display vs schema_name
+ // formatting differences. Drop the alias when it matches.
+ if expr.schema_name().to_string() == name {
Review Comment:
Is there a defined contract for metadata preservation through alias rewrites?
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -155,12 +685,494 @@ impl OptimizerRule for PushDownLeafProjections {
fn rewrite(
&self,
plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
+ config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
- Ok(Transformed::no(plan))
+ let alias_generator = config.alias_generator();
+ match try_push_input(&plan, alias_generator)? {
+ Some(new_plan) => Ok(Transformed::yes(new_plan)),
+ None => Ok(Transformed::no(plan)),
+ }
+ }
+}
+
+/// Attempts to push a projection's extractable expressions further down.
+///
+/// Returns `Some(new_subtree)` if the projection was pushed down or merged,
+/// `None` if there is nothing to push or the projection sits above a barrier.
+fn try_push_input(
+ input: &LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Option<LogicalPlan>> {
+ let LogicalPlan::Projection(proj) = input else {
+ return Ok(None);
+ };
+ split_and_push_projection(proj, alias_generator)
+}
+
+/// Splits a projection into extractable pieces, pushes them towards leaf
+/// nodes, and adds a recovery projection if needed.
+///
+/// Handles both:
+/// - **Pure extraction projections** (all `__datafusion_extracted` aliases + columns)
+/// - **Mixed projections** (containing `MoveTowardsLeafNodes` sub-expressions)
+///
+/// Returns `Some(new_subtree)` if extractions were pushed down,
+/// `None` if there is nothing to extract or push.
+///
+/// # Example: Mixed Projection
+///
+/// ```text
+/// Input plan:
+/// Projection: user['name'] IS NOT NULL AS has_name, id
+/// Filter: ...
+/// TableScan
+///
+/// Phase 1 (Split):
+/// extraction_pairs: [(user['name'], "__datafusion_extracted_1")]
+/// recovery_exprs: [__datafusion_extracted_1 IS NOT NULL AS has_name, id]
+///
+/// Phase 2 (Push):
+/// Push extraction projection through Filter toward TableScan
+///
+/// Phase 3 (Recovery):
+/// Projection: __datafusion_extracted_1 IS NOT NULL AS has_name, id <-- recovery
+/// Filter: ...
+/// Projection: user['name'] AS __datafusion_extracted_1, id <-- extraction (pushed)
+/// TableScan
+/// ```
+fn split_and_push_projection(
+ proj: &Projection,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Option<LogicalPlan>> {
+ let input = &proj.input;
+ let input_schema = input.schema();
+
+ // ── Phase 1: Split ──────────────────────────────────────────────────
+ // For each projection expression, collect extraction pairs and build
+ // recovery expressions.
+ //
+ // Pre-existing `__datafusion_extracted` aliases are inserted into the
+ // extractor's `IndexMap` with the **full** `Expr::Alias(…)` as the key,
+ // so the alias name participates in equality. This prevents collisions
+ // when CSE rewrites produce the same inner expression under different
+ // alias names (e.g. `__common_expr_4 AS __datafusion_extracted_1` and
+ // `__common_expr_4 AS __datafusion_extracted_3`). New extractions from
+ // `routing_extract` use bare (non-Alias) keys and get normal dedup.
+ //
+ // When building the final `extraction_pairs`, the Alias wrapper is
+ // stripped so consumers see the usual `(inner_expr, alias_name)` tuples.
+
+ let mut extractors = vec![LeafExpressionExtractor::new(
+ input_schema.as_ref(),
+ alias_generator,
+ )];
+ let input_column_sets = vec![schema_columns(input_schema.as_ref())];
+
+ let original_schema = proj.schema.as_ref();
+ let mut recovery_exprs: Vec<Expr> = Vec::with_capacity(proj.expr.len());
+ let mut needs_recovery = false;
+ let mut has_new_extractions = false;
+ let mut proj_exprs_captured: usize = 0;
+ // Track standalone column expressions (Case B) to detect column refs
+ // from extracted aliases (Case A) that aren't also standalone expressions.
+ let mut standalone_columns: IndexSet<Column> = IndexSet::new();
+
+ for (expr, (qualifier, field)) in proj.expr.iter().zip(original_schema.iter()) {
+ if let Expr::Alias(alias) = expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
Review Comment:
same prefix collision here, but different impact- user aliases get treated as pre existing extracted expressions and pushed down as if they were optimizer generated.
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -95,9 +101,533 @@ impl OptimizerRule for ExtractLeafExpressions {
fn rewrite(
&self,
plan: LogicalPlan,
- _config: &dyn OptimizerConfig,
+ config: &dyn OptimizerConfig,
) -> Result<Transformed<LogicalPlan>> {
- Ok(Transformed::no(plan))
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// Works for any number of inputs (0, 1, 2, …N). For multi-input nodes
+/// like Join, each extracted sub-expression is routed to the correct input
+/// by checking which input's schema contains all of the expression's column
+/// references.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Only extract from plan types whose output schema is predictable after
+ // expression rewriting. Nodes like Window derive column names from
+ // their expressions, so rewriting `get_field` inside a window function
+ // changes the output schema and breaks the recovery projection.
+ if !matches!(
+ &plan,
+ LogicalPlan::Aggregate(_)
+ | LogicalPlan::Filter(_)
+ | LogicalPlan::Sort(_)
+ | LogicalPlan::Limit(_)
+ | LogicalPlan::Join(_)
+ ) {
+ return Ok(Transformed::no(plan));
+ }
+
+ let inputs = plan.inputs();
+ if inputs.is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ // Save original output schema before any transformation
+ let original_schema = Arc::clone(plan.schema());
+
+ // Build per-input schemas from borrowed inputs (before plan is consumed
+ // by map_expressions). We only need schemas and column sets for routing;
+ // the actual inputs are cloned later only if extraction succeeds.
+ let input_schemas: Vec<Arc<DFSchema>> =
+ inputs.iter().map(|i| Arc::clone(i.schema())).collect();
+
+ // Build per-input extractors
+ let mut extractors: Vec<LeafExpressionExtractor> = input_schemas
+ .iter()
+ .map(|schema| LeafExpressionExtractor::new(schema.as_ref(), alias_generator))
+ .collect();
+
+ // Build per-input column sets for routing expressions to the correct input
+ let input_column_sets: Vec<std::collections::HashSet<Column>> = input_schemas
+ .iter()
+ .map(|schema| schema_columns(schema.as_ref()))
+ .collect();
+
+ // Transform expressions via map_expressions with routing
+ let transformed = plan.map_expressions(|expr| {
+ routing_extract(expr, &mut extractors, &input_column_sets)
+ })?;
+
+ // If no expressions were rewritten, nothing was extracted
+ if !transformed.transformed {
+ return Ok(transformed);
+ }
+
+ // Clone inputs now that we know extraction succeeded. Wrap in Arc
+ // upfront since build_extraction_projection expects &Arc<LogicalPlan>.
+ let owned_inputs: Vec<Arc<LogicalPlan>> = transformed
+ .data
+ .inputs()
+ .into_iter()
+ .map(|i| Arc::new(i.clone()))
+ .collect();
+
+ // Build per-input extraction projections (None means no extractions for that input)
+ let new_inputs: Vec<LogicalPlan> = owned_inputs
+ .into_iter()
+ .zip(extractors.iter())
+ .map(|(input_arc, extractor)| {
+ match extractor.build_extraction_projection(&input_arc)? {
+ Some(plan) => Ok(plan),
+ // No extractions for this input — recover the LogicalPlan
+ // without cloning (refcount is 1 since build returned None).
+ None => {
+ Ok(Arc::try_unwrap(input_arc).unwrap_or_else(|arc| (*arc).clone()))
+ }
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ // Rebuild the plan keeping its rewritten expressions but replacing
+ // inputs with the new extraction projections.
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Add recovery projection if the output schema changed
+ let recovered = build_recovery_projection(original_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Given an expression, returns the index of the input whose columns fully
+/// cover the expression's column references.
+/// Returns `None` if the expression references columns from multiple inputs
+/// or if multiple inputs match (ambiguous, e.g. unqualified columns present
+/// in both sides of a join).
+fn find_owning_input(
+ expr: &Expr,
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Option<usize> {
+ let mut found = None;
+ for (idx, cols) in input_column_sets.iter().enumerate() {
+ if has_all_column_refs(expr, cols) {
+ if found.is_some() {
+ // Ambiguous — multiple inputs match
+ return None;
+ }
+ found = Some(idx);
+ }
+ }
+ found
+}
+
+/// Walks an expression tree top-down, extracting `MoveTowardsLeafNodes`
+/// sub-expressions and routing each to the correct per-input extractor.
+fn routing_extract(
+ expr: Expr,
+ extractors: &mut [LeafExpressionExtractor],
+ input_column_sets: &[std::collections::HashSet<Column>],
+) -> Result<Transformed<Expr>> {
+ expr.transform_down(|e| {
+ // Skip expressions already aliased with extracted expression pattern
+ if let Expr::Alias(alias) = &e
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ {
+ return Ok(Transformed {
+ data: e,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
Review Comment:
`__datafusion_extracted*` detection is string prefix based. A user alias with that prefix will trigger `TreeNodeRecursion::Jump` and skip subtree traversal. It's worth documenting the prefix as reserved, or gating on metadata instead of the name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2760298530
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1295,6 +1297,18 @@ fn rewrite_projection(
predicates: Vec<Expr>,
mut projection: Projection,
) -> Result<(Transformed<LogicalPlan>, Option<Expr>)> {
+ // Note: This check coordinates with ExtractLeafExpressions optimizer rule.
+ // See extract_leaf_expressions.rs for details on why these projections exist.
+ // Don't push filters through extracted expression projections.
+ // Pushing filters through would rewrite expressions like `__datafusion_extracted_1 > 150`
+ // back to `get_field(s,'value') > 150`, undoing the extraction.
+ if is_extracted_expr_projection(&projection) {
Review Comment:
This is obviously not great, but I don't see another way to avoid this. Otherwise if we have:
```
Filter: get_field(col, 'foo') > 1
TableScan: projection=[col]
```
And we run our new rule to get:
```
Projection: col('col')
Filter: __datafusion_extracted_1 > 1
Projection: get_field(col, 'foo') as __datafusion_extracted_1, col
TableScan: projection=[col]
```
Then this rule runs and will produce:
```
Projection: col('col')
Projection: get_field(col, 'foo') as __datafusion_extracted_1, col
Filter: get_field(col, 'foo') > 1
TableScan projection=[col]
```
Because it wants to push the filter under the projection.
I'd argue as a general rule there's no point in pushing a filter under a projection that is purely column selections / get_field expressions especially if we can't then push it further.
Maybe a more robust fix would be to have the filter pushdown optimizer rule traverse the rest of the plan, find the position it plans to push into and then check if there's any advantage to doing some (i.e. is it pushing the filter under an expensive operator that benefits from less input data, or is it just doing a trivial pointless pushdown like in the case above). But that would be a lot more involved so I chose this simpler solution for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3844193578
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group get-field-pushdown-try-3 main
----- ------------------------ ----
logical_aggregate_with_join 1.00 645.5±4.79µs ? ?/sec 1.01 650.9±30.77µs ? ?/sec
logical_select_all_from_1000 1.00 10.3±0.08ms ? ?/sec 1.12 11.6±0.31ms ? ?/sec
logical_select_one_from_700 1.01 421.8±3.21µs ? ?/sec 1.00 419.6±8.67µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 380.8±12.06µs ? ?/sec 1.00 378.2±7.50µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.02 369.0±24.49µs ? ?/sec 1.00 362.5±4.04µs ? ?/sec
physical_intersection 1.02 1633.7±27.83µs ? ?/sec 1.00 1601.4±29.28µs ? ?/sec
physical_join_consider_sort 1.02 2.3±0.04ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_join_distinct 1.03 360.7±3.20µs ? ?/sec 1.00 351.4±2.61µs ? ?/sec
physical_many_self_joins 1.01 12.8±0.09ms ? ?/sec 1.00 12.7±0.23ms ? ?/sec
physical_plan_clickbench_all 1.02 205.6±2.81ms ? ?/sec 1.00 200.7±2.24ms ? ?/sec
physical_plan_clickbench_q1 1.02 2.2±0.03ms ? ?/sec 1.00 2.1±0.03ms ? ?/sec
physical_plan_clickbench_q10 1.02 3.7±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.11 4.6±0.18ms ? ?/sec 1.00 4.1±0.10ms ? ?/sec
physical_plan_clickbench_q12 1.04 4.4±0.08ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.03 3.8±0.07ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.03 4.2±0.07ms ? ?/sec 1.00 4.1±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.02 3.9±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q16 1.01 3.7±0.06ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q17 1.02 3.9±0.07ms ? ?/sec 1.00 3.8±0.08ms ? ?/sec
physical_plan_clickbench_q18 1.01 2.7±0.03ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q19 1.05 4.4±0.09ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q2 1.06 3.0±0.06ms ? ?/sec 1.00 2.8±0.05ms ? ?/sec
physical_plan_clickbench_q20 1.05 2.3±0.04ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.02 2.8±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q22 1.02 4.0±0.05ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.02 4.3±0.12ms ? ?/sec 1.00 4.2±0.12ms ? ?/sec
physical_plan_clickbench_q24 1.03 4.9±0.08ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.02 3.5±0.03ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
physical_plan_clickbench_q26 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.02 3.6±0.05ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.04 4.6±0.19ms ? ?/sec 1.00 4.5±0.11ms ? ?/sec
physical_plan_clickbench_q29 1.02 4.8±0.07ms ? ?/sec 1.00 4.8±0.19ms ? ?/sec
physical_plan_clickbench_q3 1.02 2.6±0.07ms ? ?/sec 1.00 2.5±0.04ms ? ?/sec
physical_plan_clickbench_q30 1.03 16.1±0.23ms ? ?/sec 1.00 15.7±0.41ms ? ?/sec
physical_plan_clickbench_q31 1.02 4.6±0.03ms ? ?/sec 1.00 4.5±0.11ms ? ?/sec
physical_plan_clickbench_q32 1.02 4.5±0.08ms ? ?/sec 1.00 4.5±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.02 3.7±0.04ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.01 3.3±0.03ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.02 3.4±0.07ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_clickbench_q36 1.03 4.3±0.11ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q37 1.01 4.8±0.05ms ? ?/sec 1.00 4.7±0.12ms ? ?/sec
physical_plan_clickbench_q38 1.01 4.7±0.08ms ? ?/sec 1.00 4.7±0.05ms ? ?/sec
physical_plan_clickbench_q39 1.01 4.1±0.15ms ? ?/sec 1.00 4.1±0.09ms ? ?/sec
physical_plan_clickbench_q4 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
physical_plan_clickbench_q40 1.02 5.0±0.04ms ? ?/sec 1.00 4.9±0.05ms ? ?/sec
physical_plan_clickbench_q41 1.01 4.3±0.05ms ? ?/sec 1.00 4.3±0.05ms ? ?/sec
physical_plan_clickbench_q42 1.02 4.3±0.09ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q43 1.02 4.6±0.10ms ? ?/sec 1.00 4.5±0.05ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.3±0.03ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q45 1.02 2.3±0.04ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.02 3.3±0.03ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.04 4.9±0.08ms ? ?/sec 1.00 4.7±0.05ms ? ?/sec
physical_plan_clickbench_q48 1.03 5.3±0.07ms ? ?/sec 1.00 5.2±0.07ms ? ?/sec
physical_plan_clickbench_q49 1.01 5.5±0.13ms ? ?/sec 1.00 5.5±0.08ms ? ?/sec
physical_plan_clickbench_q5 1.02 2.6±0.03ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q50 1.00 4.2±0.03ms ? ?/sec 1.00 4.2±0.12ms ? ?/sec
physical_plan_clickbench_q51 1.02 3.6±0.04ms ? ?/sec 1.00 3.5±0.04ms ? ?/sec
physical_plan_clickbench_q6 1.02 2.6±0.03ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.01 2.1±0.02ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.02 3.5±0.05ms ? ?/sec 1.00 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.6±0.04ms ? ?/sec 1.00 3.6±0.10ms ? ?/sec
physical_plan_tpcds_all 1.03 1985.3±26.56ms ? ?/sec 1.00 1934.2±12.37ms ? ?/sec
physical_plan_tpch_all 1.02 130.3±2.40ms ? ?/sec 1.00 127.4±1.33ms ? ?/sec
physical_plan_tpch_q1 1.08 3.2±0.07ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
physical_plan_tpch_q10 1.08 7.9±0.11ms ? ?/sec 1.00 7.3±0.08ms ? ?/sec
physical_plan_tpch_q11 1.04 8.9±0.09ms ? ?/sec 1.00 8.6±0.14ms ? ?/sec
physical_plan_tpch_q12 1.03 3.2±0.05ms ? ?/sec 1.00 3.1±0.06ms ? ?/sec
physical_plan_tpch_q13 1.04 3.2±0.15ms ? ?/sec 1.00 3.0±0.07ms ? ?/sec
physical_plan_tpch_q14 1.03 3.3±0.08ms ? ?/sec 1.00 3.2±0.07ms ? ?/sec
physical_plan_tpch_q16 1.04 5.4±0.17ms ? ?/sec 1.00 5.2±0.04ms ? ?/sec
physical_plan_tpch_q17 1.05 5.9±0.18ms ? ?/sec 1.00 5.7±0.05ms ? ?/sec
physical_plan_tpch_q18 1.04 6.2±0.11ms ? ?/sec 1.00 6.0±0.08ms ? ?/sec
physical_plan_tpch_q19 1.06 5.3±0.09ms ? ?/sec 1.00 5.1±0.07ms ? ?/sec
physical_plan_tpch_q2 1.05 13.0±0.16ms ? ?/sec 1.00 12.4±0.13ms ? ?/sec
physical_plan_tpch_q20 1.10 8.8±0.11ms ? ?/sec 1.00 8.0±0.05ms ? ?/sec
physical_plan_tpch_q21 1.05 10.6±0.20ms ? ?/sec 1.00 10.2±0.16ms ? ?/sec
physical_plan_tpch_q22 1.03 6.6±0.07ms ? ?/sec 1.00 6.4±0.07ms ? ?/sec
physical_plan_tpch_q3 1.04 5.9±0.10ms ? ?/sec 1.00 5.6±0.14ms ? ?/sec
physical_plan_tpch_q4 1.03 3.1±0.05ms ? ?/sec 1.00 3.0±0.09ms ? ?/sec
physical_plan_tpch_q5 1.03 6.2±0.06ms ? ?/sec 1.00 6.0±0.04ms ? ?/sec
physical_plan_tpch_q6 1.03 1652.5±25.45µs ? ?/sec 1.00 1598.0±18.11µs ? ?/sec
physical_plan_tpch_q7 1.03 7.5±0.16ms ? ?/sec 1.00 7.2±0.04ms ? ?/sec
physical_plan_tpch_q8 1.04 9.8±0.11ms ? ?/sec 1.00 9.4±0.06ms ? ?/sec
physical_plan_tpch_q9 1.05 7.0±0.09ms ? ?/sec 1.00 6.7±0.07ms ? ?/sec
physical_select_aggregates_from_200 1.00 17.7±0.14ms ? ?/sec 1.01 17.9±0.20ms ? ?/sec
physical_select_all_from_1000 1.00 23.6±0.18ms ? ?/sec 1.06 25.1±0.22ms ? ?/sec
physical_select_one_from_700 1.02 1360.2±32.81µs ? ?/sec 1.00 1337.9±14.27µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.03 11.4±0.44ms ? ?/sec 1.00 11.1±0.05ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.04 31.0±0.23ms ? ?/sec 1.00 29.9±0.23ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.02 201.8±2.06ms ? ?/sec 1.00 197.7±2.07ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.06 1164.9±18.35ms ? ?/sec 1.00 1095.4±9.10ms ? ?/sec
physical_theta_join_consider_sort 1.08 2.9±0.06ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_unnest_to_join 1.05 3.2±0.05ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_window_function_partition_by_12_on_values 1.01 1623.0±34.06µs ? ?/sec 1.00 1602.8±26.71µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.01 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 1103.3±7.88µs ? ?/sec 1.00 1106.5±18.21µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 1282.8±10.81µs ? ?/sec 1.00 1279.9±13.22µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.01 1370.3±24.28µs ? ?/sec 1.00 1350.5±11.53µs ? ?/sec
with_param_values_many_columns 1.00 575.5±4.77µs ? ?/sec 1.05 602.3±6.73µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843700426
> > @alamb are there any of these that use structs? It seems like this has no impact on the benchmarks (good!) but maybe we should add some that hit the full rewrite?
>
> I was more trying to quantify the effect on planning time of adding a new optimizer pass -- it seems like it is small but noticable slowdown (1-3%). I'll see if I can reproduce those results
It makes sense that there’s a small slowdown, it has to visit every node in the plan even if it doesn’t modify it at all. That said a lot of the numbers were within the variation ie not statistically different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893119496
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3893708080
Thank you @ethan-tyler and @alamb for the review.
Andrew I am happy to help efforts to improve performance of logical optimizer. Like we've discussed on calls the solution is likely "big wins" like making the optimizer runs copy on write instead of always copy. Other solutions like micro-optimizing individual optimizer rules or trying to reduce optimizer rules are likely to have a negative tradeoff in terms of code complexity or functionality.
I plan to merge this once CI passes since it's relatively self contained (should not disrupt other users and doesn't change any public APIs) and has been open for several days in nearly the same state. I'm happy to address any further feedback in follow up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756607559
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1870 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing extracted
+//! expression projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs, is_extracted_expr_projection};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+ LogicalPlan::Join(_) => extract_from_join(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
Review Comment:
I'm not sure what else we could handle here. Maybe Extension?
Before we merge this PR we expand this to explicitly ignore all other nodes so that if a new node is added one has to decide how this rule should handle it. I'll wait to do that since that's another +30 LOC diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756630900
##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -168,7 +168,7 @@ query TT
EXPLAIN SELECT id, s['value'] + 1 FROM simple_struct;
----
logical_plan
-01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1)
+01)Projection: simple_struct.id, get_field(simple_struct.s, Utf8("value")) + Int64(1) AS simple_struct.s[value] + Int64(1)
Review Comment:
Note that this is not a change in the output schema name: it is already `simple_struct.s[value]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#issuecomment-3843742066
> It makes sense that there’s a small slowdown, it has to visit every node in the plan even if it doesn’t modify it at all. That said a lot of the numbers were within the variation ie not statistically different.
Yeah, there is a tension here for sure
I do think in general it would be a good idea to consolidate optimizer passes given each basically deep copies the plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2756235399
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1291,10 +1291,38 @@ impl OptimizerRule for PushDownFilter {
/// Filter(foo=5)
/// ...
/// ```
+/// Check if a projection is a `__leaf_*` extraction projection
+/// (created by ExtractLeafExpressions).
+///
+/// These projections should not have filters pushed through them because doing so
+/// would rewrite the filter expressions back to their original form (e.g., rewriting
+/// `__leaf_1 > 150` back to `get_field(s,'value') > 150`), which undoes the extraction
+/// and prevents proper pushdown of field access expressions.
+fn is_leaf_extraction_projection(proj: &Projection) -> bool {
+ proj.expr.iter().any(|e| {
+ if let Expr::Alias(alias) = e {
+ alias.name.starts_with("__leaf")
Review Comment:
The magic string `__leaf` is duplicated in multiple locations throughout the codebase. Consider defining it as a constant (e.g., `const LEAF_ALIAS_PREFIX: &str = \"__leaf\";`) to ensure consistency and make future changes easier.
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1059 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **BottomUp** traversal to push ALL `MoveTowardsLeafNodes` expressions
+//! (like `get_field`) to projections immediately above scan nodes. This enables optimal
+//! Parquet column pruning.
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//! - Passthrough `Projection` - only column references
+//!
+//! ### How It Works
+//!
+//! 1. Process leaf nodes first (TableScan, etc.)
+//! 2. When processing higher nodes, descendants are already finalized
+//! 3. Push extractions DOWN through the plan, merging into existing `__leaf_*`
+//! projections when possible
+
+use indexmap::{IndexMap, IndexSet};
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __leaf_1 = 'active'
+/// Projection: user['status'] AS __leaf_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+#[derive(Default, Debug)]
+pub struct ExtractLeafExpressions {}
+
+impl ExtractLeafExpressions {
+ /// Create a new [`ExtractLeafExpressions`]
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for ExtractLeafExpressions {
+ fn name(&self) -> &str {
+ "extract_leaf_expressions"
+ }
+
+ fn apply_order(&self) -> Option<ApplyOrder> {
+ Some(ApplyOrder::BottomUp)
+ }
+
+ fn rewrite(
+ &self,
+ plan: LogicalPlan,
+ config: &dyn OptimizerConfig,
+ ) -> Result<Transformed<LogicalPlan>> {
+ let alias_generator = config.alias_generator();
+ extract_from_plan(plan, alias_generator)
+ }
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from a plan node.
+///
+/// With BottomUp traversal, we process leaves first, then work up.
+/// This allows us to push extractions all the way down to scan nodes.
+fn extract_from_plan(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ match &plan {
+ // Schema-preserving nodes - extract and push down
+ LogicalPlan::Filter(_) | LogicalPlan::Sort(_) | LogicalPlan::Limit(_) => {
+ extract_from_schema_preserving(plan, alias_generator)
+ }
+
+ // Schema-transforming nodes need special handling
+ LogicalPlan::Aggregate(_) => extract_from_aggregate(plan, alias_generator),
+ LogicalPlan::Projection(_) => extract_from_projection(plan, alias_generator),
+
+ // Everything else passes through unchanged
+ _ => Ok(Transformed::no(plan)),
+ }
+}
+
+/// Extracts from schema-preserving nodes (Filter, Sort, Limit).
+///
+/// These nodes don't change the schema, so we can extract expressions
+/// and push them down to existing leaf projections or create new ones.
+///
+/// Uses CSE's two-level pattern:
+/// 1. Inner extraction projection with ALL columns passed through
+/// 2. Outer recovery projection to restore original schema
+fn extract_from_schema_preserving(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ // Skip nodes with no children
+ if plan.inputs().is_empty() {
+ return Ok(Transformed::no(plan));
+ }
+
+ let input = plan.inputs()[0].clone();
+ let input_schema = Arc::clone(input.schema());
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let input_arc = Arc::new(input);
+ let (target, path) = find_extraction_target(&input_arc);
+ let target_schema = Arc::clone(target.schema());
+
+ // Extract using target schema - this is where the projection will be placed
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Transform expressions
+ let transformed = plan.map_expressions(|expr| extractor.extract(expr))?;
+
+ if !extractor.has_extractions() {
+ return Ok(transformed);
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_leaf_projection(&target) {
+ merge_into_leaf_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild the path from target back up to our node's input
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create the node with new input
+ let new_inputs: Vec<LogicalPlan> = std::iter::once(rebuilt_input)
+ .chain(
+ transformed
+ .data
+ .inputs()
+ .iter()
+ .skip(1)
+ .map(|p| (*p).clone()),
+ )
+ .collect();
+
+ let new_plan = transformed
+ .data
+ .with_new_exprs(transformed.data.expressions(), new_inputs)?;
+
+ // Use CSE's pattern: add recovery projection to restore original schema
+ let recovered = build_recover_project_plan(input_schema.as_ref(), new_plan)?;
+
+ Ok(Transformed::yes(recovered))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Aggregate nodes.
+///
+/// For Aggregates, we extract from:
+/// - Group-by expressions (full expressions or sub-expressions)
+/// - Arguments inside aggregate functions (NOT the aggregate function itself)
+///
+/// Uses CSE's two-level pattern with NamePreserver for stable name handling.
+fn extract_from_aggregate(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Aggregate(agg) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_group_names: Vec<_> = agg
+ .group_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+ let saved_aggr_names: Vec<_> = agg
+ .aggr_expr
+ .iter()
+ .map(|e| name_preserver.save(e))
+ .collect();
+
+ // Find where to place extractions
+ let (target, path) = find_extraction_target(&agg.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from group-by expressions
+ let mut new_group_by = Vec::with_capacity(agg.group_expr.len());
+ let mut has_extractions = false;
+
+ for expr in &agg.group_expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_group_by.push(transformed.data);
+ }
+
+ // Extract from aggregate function arguments (not the function itself)
+ let mut new_aggr = Vec::with_capacity(agg.aggr_expr.len());
+
+ for expr in &agg.aggr_expr {
+ let transformed = extract_from_aggregate_args(expr.clone(), &mut extractor)?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_aggr.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Aggregate(agg)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_leaf_projection(&target) {
+ merge_into_leaf_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Restore names in group-by expressions using NamePreserver
+ let restored_group_expr: Vec<Expr> = new_group_by
+ .into_iter()
+ .zip(saved_group_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Restore names in aggregate expressions using NamePreserver
+ let restored_aggr_expr: Vec<Expr> = new_aggr
+ .into_iter()
+ .zip(saved_aggr_names)
+ .map(|(expr, saved)| saved.restore(expr))
+ .collect();
+
+ // Create new Aggregate with restored names
+ // (no outer projection needed if names are properly preserved)
+ let new_agg = datafusion_expr::logical_plan::Aggregate::try_new(
+ Arc::new(rebuilt_input),
+ restored_group_expr,
+ restored_aggr_expr,
+ )?;
+
+ Ok(Transformed::yes(LogicalPlan::Aggregate(new_agg)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from Projection nodes.
+///
+/// Uses CSE's two-level pattern (outer + inner projections only):
+/// - Inner projection: extraction with ALL columns passed through
+/// - Outer projection: rewritten expressions with restored names
+///
+/// This avoids the unstable 3-level structure that gets broken by OptimizeProjections.
+fn extract_from_projection(
+ plan: LogicalPlan,
+ alias_generator: &Arc<AliasGenerator>,
+) -> Result<Transformed<LogicalPlan>> {
+ let LogicalPlan::Projection(proj) = plan else {
+ return Ok(Transformed::no(plan));
+ };
+
+ // Skip if this projection is fully extracted (only column references)
+ if is_fully_extracted(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Skip if this is already a leaf projection (contains __leaf_* aliases).
+ // This prevents re-extraction on subsequent optimizer passes.
+ if is_leaf_projection(&proj) {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Save original expression names using NamePreserver (like CSE)
+ let name_preserver = NamePreserver::new_for_projection();
+ let saved_names: Vec<_> = proj.expr.iter().map(|e| name_preserver.save(e)).collect();
+
+ // Find where to place extractions (look down through schema-preserving nodes)
+ let (target, path) = find_extraction_target(&proj.input);
+ let target_schema = Arc::clone(target.schema());
+
+ let mut extractor =
+ LeafExpressionExtractor::new(target_schema.as_ref(), alias_generator);
+
+ // Extract from projection expressions
+ let mut new_exprs = Vec::with_capacity(proj.expr.len());
+ let mut has_extractions = false;
+
+ for expr in &proj.expr {
+ let transformed = extractor.extract(expr.clone())?;
+ if transformed.transformed {
+ has_extractions = true;
+ }
+ new_exprs.push(transformed.data);
+ }
+
+ if !has_extractions {
+ return Ok(Transformed::no(LogicalPlan::Projection(proj)));
+ }
+
+ // Build extraction projection with ALL columns (CSE-style)
+ let extraction_proj = if let Some(existing_proj) = get_leaf_projection(&target) {
+ merge_into_leaf_projection(existing_proj, &extractor)?
+ } else {
+ extractor.build_projection_with_all_columns(target)?
+ };
+
+ // Rebuild path from target back up
+ let rebuilt_input = rebuild_path(path, LogicalPlan::Projection(extraction_proj))?;
+
+ // Create outer projection with rewritten exprs + restored names
+ let final_exprs: Vec<Expr> = new_exprs
+ .into_iter()
+ .zip(saved_names)
+ .map(|(expr, saved_name)| saved_name.restore(expr))
+ .collect();
+
+ let outer_projection = Projection::try_new(final_exprs, Arc::new(rebuilt_input))?;
+
+ Ok(Transformed::yes(LogicalPlan::Projection(outer_projection)))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from aggregate function arguments.
+///
+/// This extracts from inside the aggregate (e.g., from `sum(get_field(x, 'y'))`
+/// we extract `get_field(x, 'y')`), but NOT the aggregate function itself.
+fn extract_from_aggregate_args(
+ expr: Expr,
+ extractor: &mut LeafExpressionExtractor,
+) -> Result<Transformed<Expr>> {
+ match expr {
+ Expr::AggregateFunction(mut agg_func) => {
+ // Extract from arguments, not the function itself
+ let mut any_changed = false;
+ let mut new_args = Vec::with_capacity(agg_func.params.args.len());
+
+ for arg in agg_func.params.args {
+ let transformed = extractor.extract(arg)?;
+ if transformed.transformed {
+ any_changed = true;
+ }
+ new_args.push(transformed.data);
+ }
+
+ if any_changed {
+ agg_func.params.args = new_args;
+ Ok(Transformed::yes(Expr::AggregateFunction(agg_func)))
+ } else {
+ agg_func.params.args = new_args;
+ Ok(Transformed::no(Expr::AggregateFunction(agg_func)))
+ }
+ }
+ // For aliased aggregates, process the inner expression
+ Expr::Alias(alias) => {
+ let transformed = extract_from_aggregate_args(*alias.expr, extractor)?;
+ Ok(
+ transformed
+ .update_data(|e| e.alias_qualified(alias.relation, alias.name)),
+ )
+ }
+ // For other expressions, use regular extraction
+ other => extractor.extract(other),
+ }
+}
+
+// =============================================================================
+// Helper Functions for BottomUp Traversal
+// =============================================================================
+
+/// Traverses down through schema-preserving nodes to find where to place extractions.
+///
+/// Returns (target_node, path_to_rebuild) where:
+/// - target_node: the node above which to create extraction projection
+/// - path_to_rebuild: nodes between our input and target that must be rebuilt
+///
+/// Schema-preserving nodes that we can look through:
+/// - Filter, Sort, Limit: pass all input columns through unchanged
+/// - Passthrough projections: only column references
+///
+/// Barrier nodes where we stop:
+/// - TableScan, Join, Aggregate: these are extraction targets
+/// - Existing __leaf_* projections: we merge into these
+/// - Any other node type
+fn find_extraction_target(
+ input: &Arc<LogicalPlan>,
+) -> (Arc<LogicalPlan>, Vec<LogicalPlan>) {
+ let mut current = Arc::clone(input);
+ let mut path = vec![];
+
+ loop {
+ match current.as_ref() {
+ // Look through schema-preserving nodes
+ LogicalPlan::Filter(f) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&f.input);
+ }
+ LogicalPlan::Sort(s) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&s.input);
+ }
+ LogicalPlan::Limit(l) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&l.input);
+ }
+ // Look through passthrough projections (only column references)
+ LogicalPlan::Projection(p) if is_passthrough_projection(p) => {
+ path.push(current.as_ref().clone());
+ current = Arc::clone(&p.input);
+ }
+ // Found existing __leaf_* projection - will merge into it
+ LogicalPlan::Projection(p) if is_leaf_projection(p) => {
+ return (current, path);
+ }
+ // Hit a barrier node - create new projection here
+ _ => {
+ return (current, path);
+ }
+ }
+ }
+}
+
+/// Returns true if the projection contains `__leaf_*` expressions (created by us).
+fn is_leaf_projection(proj: &Projection) -> bool {
+ proj.expr.iter().any(|e| {
+ if let Expr::Alias(alias) = e {
+ alias.name.starts_with("__leaf")
+ } else {
+ false
+ }
+ })
+}
+
+/// Returns true if the projection is a passthrough (only column references).
+fn is_passthrough_projection(proj: &Projection) -> bool {
+ proj.expr.iter().all(|e| matches!(e, Expr::Column(_)))
+}
+
+/// Returns true if the projection only has column references (nothing to extract).
+fn is_fully_extracted(proj: &Projection) -> bool {
+ proj.expr.iter().all(|e| {
+ matches!(e, Expr::Column(_))
+ || matches!(e, Expr::Alias(a) if matches!(a.expr.as_ref(), Expr::Column(_)))
+ })
+}
+
+/// If the target is a leaf projection, return it for merging.
+fn get_leaf_projection(target: &Arc<LogicalPlan>) -> Option<&Projection> {
+ if let LogicalPlan::Projection(p) = target.as_ref()
+ && is_leaf_projection(p)
+ {
+ return Some(p);
+ }
+ None
+}
+
+/// Merges new extractions into an existing __leaf_* projection.
+fn merge_into_leaf_projection(
+ existing: &Projection,
+ extractor: &LeafExpressionExtractor,
+) -> Result<Projection> {
+ let mut proj_exprs = existing.expr.clone();
+
+ // Build a map of existing expressions (by schema_name) to their aliases
+ let existing_extractions: IndexMap<String, String> = existing
+ .expr
+ .iter()
+ .filter_map(|e| {
+ if let Expr::Alias(alias) = e
+ && alias.name.starts_with("__leaf")
+ {
+ let schema_name = alias.expr.schema_name().to_string();
+ return Some((schema_name, alias.name.clone()));
+ }
+ None
+ })
+ .collect();
+
+ // Add new extracted expressions, but only if not already present
+ for (schema_name, (expr, alias)) in &extractor.extracted {
+ if !existing_extractions.contains_key(schema_name) {
+ proj_exprs.push(expr.clone().alias(alias));
+ }
+ }
+
+ // Add any new pass-through columns that aren't already in the projection
+ let existing_cols: IndexSet<Column> = existing
+ .expr
+ .iter()
+ .filter_map(|e| {
+ if let Expr::Column(c) = e {
+ Some(c.clone())
+ } else {
+ None
+ }
+ })
+ .collect();
+
+ for col in &extractor.columns_needed {
+ if !existing_cols.contains(col) && extractor.input_schema.has_column(col) {
+ proj_exprs.push(Expr::Column(col.clone()));
+ }
+ }
+
+ Projection::try_new(proj_exprs, Arc::clone(&existing.input))
+}
+
+/// Rebuilds the path from extraction projection back up to original input.
+///
+/// Takes a list of nodes (in top-to-bottom order from input towards target)
+/// and rebuilds them with the new bottom input.
+///
+/// For passthrough projections, we update them to include ALL columns from
+/// the new input (including any new `__leaf_*` columns that were merged).
+fn rebuild_path(path: Vec<LogicalPlan>, new_bottom: LogicalPlan) -> Result<LogicalPlan> {
+ let mut current = new_bottom;
+
+ // Rebuild path from bottom to top (reverse order)
+ for node in path.into_iter().rev() {
+ current = match node {
+ LogicalPlan::Filter(f) => {
+ LogicalPlan::Filter(Filter::try_new(f.predicate, Arc::new(current))?)
+ }
+ LogicalPlan::Sort(s) => LogicalPlan::Sort(Sort {
+ expr: s.expr,
+ input: Arc::new(current),
+ fetch: s.fetch,
+ }),
+ LogicalPlan::Limit(l) => LogicalPlan::Limit(Limit {
+ skip: l.skip,
+ fetch: l.fetch,
+ input: Arc::new(current),
+ }),
+ LogicalPlan::Projection(p) if is_passthrough_projection(&p) => {
+ // For passthrough projections, include ALL columns from new input
+ // This ensures new __leaf_* columns flow through
+ let new_exprs: Vec<Expr> = current
+ .schema()
+ .columns()
+ .into_iter()
+ .map(Expr::Column)
+ .collect();
+ LogicalPlan::Projection(Projection::try_new(
+ new_exprs,
+ Arc::new(current),
+ )?)
+ }
+ LogicalPlan::Projection(p) => {
+ LogicalPlan::Projection(Projection::try_new(p.expr, Arc::new(current))?)
+ }
+ // Should not happen based on find_extraction_target, but handle gracefully
+ other => other.with_new_exprs(other.expressions(), vec![current])?,
+ };
+ }
+
+ Ok(current)
+}
+
+/// Build projection to restore original schema (like CSE's build_recover_project_plan).
+///
+/// This adds a projection that selects only the columns from the original schema,
+/// hiding any intermediate `__leaf_*` columns that were added during extraction.
+fn build_recover_project_plan(
+ schema: &DFSchema,
+ input: LogicalPlan,
+) -> Result<LogicalPlan> {
+ let col_exprs: Vec<Expr> = schema.iter().map(Expr::from).collect();
+ let projection = Projection::try_new(col_exprs, Arc::new(input))?;
+ Ok(LogicalPlan::Projection(projection))
+}
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from larger expressions.
+struct LeafExpressionExtractor<'a> {
+ /// Extracted expressions: maps schema_name -> (original_expr, alias)
+ extracted: IndexMap<String, (Expr, String)>,
+ /// Columns needed for pass-through
+ columns_needed: IndexSet<Column>,
+ /// Input schema
+ input_schema: &'a DFSchema,
+ /// Alias generator
+ alias_generator: &'a Arc<AliasGenerator>,
+}
+
+impl<'a> LeafExpressionExtractor<'a> {
+ fn new(input_schema: &'a DFSchema, alias_generator: &'a Arc<AliasGenerator>) -> Self {
+ Self {
+ extracted: IndexMap::new(),
+ columns_needed: IndexSet::new(),
+ input_schema,
+ alias_generator,
+ }
+ }
+
+ /// Extracts `MoveTowardsLeafNodes` sub-expressions, returning rewritten expression.
+ fn extract(&mut self, expr: Expr) -> Result<Transformed<Expr>> {
+ // Walk top-down to find MoveTowardsLeafNodes sub-expressions
+ expr.transform_down(|e| {
+ // Skip expressions already aliased with __leaf_* pattern.
+ // These were created by a previous extraction pass and should not be
+ // extracted again. Use TreeNodeRecursion::Jump to skip children.
+ if let Expr::Alias(alias) = &e
+ && alias.name.starts_with("__leaf")
+ {
+ return Ok(Transformed {
+ data: e,
+ transformed: false,
+ tnr: TreeNodeRecursion::Jump,
+ });
+ }
+
+ match e.placement() {
+ ExpressionPlacement::MoveTowardsLeafNodes => {
+ // Extract this entire sub-tree
+ let col_ref = self.add_extracted(e)?;
+ Ok(Transformed::yes(col_ref))
+ }
+ ExpressionPlacement::Column => {
+ // Track columns for pass-through
+ if let Expr::Column(col) = &e {
+ self.columns_needed.insert(col.clone());
+ }
+ Ok(Transformed::no(e))
+ }
+ _ => {
+ // Continue recursing into children
+ Ok(Transformed::no(e))
+ }
+ }
+ })
+ }
+
+ /// Adds an expression to extracted set, returns column reference.
+ fn add_extracted(&mut self, expr: Expr) -> Result<Expr> {
+ let schema_name = expr.schema_name().to_string();
+
+ // Deduplication: reuse existing alias if same expression
+ if let Some((_, alias)) = self.extracted.get(&schema_name) {
+ return Ok(Expr::Column(Column::new_unqualified(alias)));
+ }
+
+ // Track columns referenced by this expression
+ for col in expr.column_refs() {
+ self.columns_needed.insert(col.clone());
+ }
+
+ // Generate unique alias
+ let alias = self.alias_generator.next("__leaf");
Review Comment:
The magic string `__leaf` is used here and in multiple other places. Consider defining it as a module-level or crate-level constant to centralize this value and make it easier to change if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add ExtractLeafExpressions optimizer rule for get_field pushdown [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20117:
URL: https://github.com/apache/datafusion/pull/20117#discussion_r2782411805
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -0,0 +1,1916 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`ExtractLeafExpressions`] extracts `MoveTowardsLeafNodes` sub-expressions into projections.
+//!
+//! This optimizer rule normalizes the plan so that all `MoveTowardsLeafNodes` computations
+//! (like field accessors) live in Projection nodes immediately above scan nodes, making them
+//! eligible for pushdown by the `OptimizeProjections` rule.
+//!
+//! ## Algorithm
+//!
+//! This rule uses **TopDown** traversal with projection merging:
+//!
+//! 1. When encountering a projection with `MoveTowardsLeafNodes` expressions, look at its input
+//! 2. If input is a Projection, **merge** the expressions through it using column replacement
+//! 3. Continue until we hit a barrier node (TableScan, Join, Aggregate)
+//! 4. Idempotency is natural: merged expressions no longer have column refs matching projection outputs
+//!
+//! ### Special Cases
+//!
+//! - If ALL expressions in a projection are `MoveTowardsLeafNodes`, push the entire projection down
+//! - If NO expressions are `MoveTowardsLeafNodes`, return `Transformed::no`
+//!
+//! ### Node Classification
+//!
+//! **Barrier Nodes** (stop pushing, create projection above):
+//! - `TableScan` - the leaf, ideal extraction point
+//! - `Join` - requires routing to left/right sides
+//! - `Aggregate` - changes schema semantics
+//! - `SubqueryAlias` - scope boundary
+//! - `Union`, `Intersect`, `Except` - schema boundaries
+//!
+//! **Schema-Preserving Nodes** (push through unchanged):
+//! - `Filter` - passes all input columns through
+//! - `Sort` - passes all input columns through
+//! - `Limit` - passes all input columns through
+//!
+//! **Projection Nodes** (merge through):
+//! - Replace column refs with underlying expressions from the child projection
+
+use indexmap::{IndexMap, IndexSet};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion_common::alias::AliasGenerator;
+use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
+use datafusion_common::{Column, DFSchema, Result};
+use datafusion_expr::expr_rewriter::NamePreserver;
+use datafusion_expr::logical_plan::LogicalPlan;
+use datafusion_expr::{Expr, ExpressionPlacement, Filter, Limit, Projection, Sort};
+
+use crate::optimizer::ApplyOrder;
+use crate::push_down_filter::replace_cols_by_name;
+use crate::utils::{EXTRACTED_EXPR_PREFIX, has_all_column_refs};
+use crate::{OptimizerConfig, OptimizerRule};
+
+/// Extracts `MoveTowardsLeafNodes` sub-expressions from all nodes into projections.
+///
+/// This normalizes the plan so that all `MoveTowardsLeafNodes` computations (like field
+/// accessors) live in Projection nodes, making them eligible for pushdown.
+///
+/// # Example
+///
+/// Given a filter with a struct field access:
+///
+/// ```text
+/// Filter: user['status'] = 'active'
+/// TableScan: t [user]
+/// ```
+///
+/// This rule extracts the field access into a projection:
+///
+/// ```text
+/// Filter: __datafusion_extracted_1 = 'active'
+/// Projection: user['status'] AS __datafusion_extracted_1, user
+/// TableScan: t [user]
+/// ```
+///
+/// The `OptimizeProjections` rule can then push this projection down to the scan.
+///
+/// **Important:** The `PushDownFilter` rule is aware of projections created by this rule
+/// and will not push filters through them. See `is_extracted_expr_projection` in utils.rs.
Review Comment:
It's `pub(crate)` so it can't be linked to...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org