You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/22 08:38:52 UTC
(datafusion) branch gh-readonly-queue/main/pr-26557-2717e25625ff0c51c1805f8df52d317979db74ec created (now 00ef9606aa)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-26557-2717e25625ff0c51c1805f8df52d317979db74ec
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 00ef9606aa fix: map remaining union input by position in `propagate_empty_relation` (#26557)
This branch includes the following new commits:
new 00ef9606aa fix: map remaining union input by position in `propagate_empty_relation` (#26557)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: map remaining union input by position in `propagate_empty_relation` (#26557)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-26557-2717e25625ff0c51c1805f8df52d317979db74ec
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 00ef9606aaa8c2f878f98daa61225380574998ad
Author: Jorge Morais <jo...@datadoghq.com>
AuthorDate: Thu Oct 22 08:38:27 2026 +0000
fix: map remaining union input by position in `propagate_empty_relation` (#26557)
## Which issue does this PR close?
- Closes #26516.
## Rationale for this change
A `UNION ALL` fails to plan with `Schema error: No field named ...` when
the optimizer removes its first input and a later input remains. For
example:
```sql
SELECT 1 WHERE false UNION ALL (SELECT 2 LIMIT 1);
```
`propagate_empty_relation` puts a projection over the remaining input
with `Projection::new_from_schema`. That function makes column
references from the union's field names, which come from the first
input. When the remaining input has other names, the projection refers
to columns that do not exist, and `optimize_projections` fails.
## What changes are included in this PR?
- When one input remains and its schema is different from the union's
schema, map its columns to the union's output fields **by position**,
alias each one to the union's name, and build the projection with
`Projection::try_new` (which also checks the result).
- Three existing snapshots change their text from `Projection: a, b, c`
to `Projection: test.a AS a, test.b AS b, test.c AS c`. In those tests
the remaining input is the first input, so the old by-name references
worked. The output schema is the same.
- This also changes the `EXPLAIN` output of the `struct.slt` test added
in #25412. That PR made `push_down_leaf_projections` handle the
bare-name projection that `propagate_empty_relation` left after it
removed an empty union input. With this change, the projection aliases
the qualified input columns (`leaf_merge_source.s AS s`). The test still
plans and returns the same rows. #25412 also handles this same-name
alias form, and its unit tests still build the bare-name shape directly.
## What is the testing strategy for this PR?
- New unit test `propagate_union_remaining_input_with_other_names`: the
first input is empty and the second input has other column names.
Without the fix, the rule produces `Projection: a, b, c` over an input
that only has `x, y, z`.
- New `union.slt` cases with the two queries from the issue.
## Are there any user-facing changes?
No. The queries that failed now return results. There are no public API
changes.
Co-authored-by: Jorge Morais <19...@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
.../optimizer/src/propagate_empty_relation.rs | 52 +++++++++++++++++++---
datafusion/sqllogictest/test_files/struct.slt | 6 +--
datafusion/sqllogictest/test_files/union.slt | 14 ++++++
3 files changed, 62 insertions(+), 10 deletions(-)
diff --git a/datafusion/optimizer/src/propagate_empty_relation.rs b/datafusion/optimizer/src/propagate_empty_relation.rs
index bd96c55861..f9183c467f 100644
--- a/datafusion/optimizer/src/propagate_empty_relation.rs
+++ b/datafusion/optimizer/src/propagate_empty_relation.rs
@@ -210,11 +210,21 @@ impl OptimizerRule for PropagateEmptyRelation {
if child.schema().eq(plan.schema()) {
Ok(Transformed::yes(child))
} else {
+ // The union takes its field names from its first
+ // input, which may be gone. Map the remaining
+ // input's columns by position.
+ let exprs = child
+ .schema()
+ .columns()
+ .into_iter()
+ .zip(plan.schema().iter())
+ .map(|(col, (qualifier, field))| {
+ Expr::Column(col)
+ .alias_qualified(qualifier.cloned(), field.name())
+ })
+ .collect::<Vec<_>>();
Ok(Transformed::yes(LogicalPlan::Projection(
- Projection::new_from_schema(
- Arc::new(child),
- Arc::clone(plan.schema()),
- ),
+ Projection::try_new(exprs, Arc::new(child))?,
)))
}
} else {
@@ -439,7 +449,8 @@ mod tests {
let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
- let expected = "Projection: a, b, c\n TableScan: test";
+ let expected =
+ "Projection: test.a AS a, test.b AS b, test.c AS c\n TableScan: test";
assert_together_optimized_plan(plan, expected, true)
}
@@ -529,7 +540,34 @@ mod tests {
let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
- let expected = "Projection: a, b, c\n TableScan: test";
+ let expected =
+ "Projection: test.a AS a, test.b AS b, test.c AS c\n TableScan: test";
+ assert_together_optimized_plan(plan, expected, true)
+ }
+
+ #[test]
+ fn propagate_union_remaining_input_with_other_names() -> Result<()> {
+ // Union
+ // Filter: false
+ // TableScan: test
+ // Projection: test2.a AS x, test2.b AS y, test2.c AS z
+ // TableScan: test2
+ let left = LogicalPlanBuilder::from(test_table_scan()?)
+ .filter(lit(false))?
+ .build()?;
+ let right = LogicalPlanBuilder::from(test_table_scan_with_name("test2")?)
+ .project(vec![
+ col("a").alias("x"),
+ col("b").alias("y"),
+ col("c").alias("z"),
+ ])?
+ .build()?;
+
+ let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
+
+ let expected = "Projection: x AS a, y AS b, z AS c\
+ \n Projection: test2.a AS x, test2.b AS y, test2.c AS z\
+ \n TableScan: test2";
assert_together_optimized_plan(plan, expected, true)
}
@@ -813,7 +851,7 @@ mod tests {
.union(three)?
.build()?;
- let expected = "Projection: a, b, c\
+ let expected = "Projection: test.a AS a, test.b AS b, test.c AS c\
\n TableScan: test";
assert_together_optimized_plan(plan, expected, true)
diff --git a/datafusion/sqllogictest/test_files/struct.slt b/datafusion/sqllogictest/test_files/struct.slt
index 6e0e1b483b..5b2ef30b20 100644
--- a/datafusion/sqllogictest/test_files/struct.slt
+++ b/datafusion/sqllogictest/test_files/struct.slt
@@ -1982,8 +1982,8 @@ statement ok
drop table rename_swap_struct_field;
# Merging an extraction projection into a projection whose output lost its
-# qualifier. Eliminating the empty side of the union leaves a projection of
-# bare column names over a qualified input, and the merge used to add the
+# qualifier. Eliminating the empty side of the union leaves a projection that
+# renames qualified input columns to bare names, and the merge used to add the
# pass-through columns under those bare names beside the qualified ones the
# projection already carried, which is an ambiguous schema.
statement ok
@@ -2009,7 +2009,7 @@ logical_plan
03)----Aggregate: groupBy=[[expanded.env]], aggr=[[sum(CAST(__datafusion_extracted_1 AS Int64))]]
04)------SubqueryAlias: expanded
05)--------SubqueryAlias: samples
-06)----------Projection: leaf_merge_source.env, get_field(s, Utf8("a")) AS __datafusion_extracted_1
+06)----------Projection: leaf_merge_source.env, get_field(leaf_merge_source.s, Utf8("a")) AS __datafusion_extracted_1
07)------------TableScan: leaf_merge_source projection=[s, env]
statement ok
diff --git a/datafusion/sqllogictest/test_files/union.slt b/datafusion/sqllogictest/test_files/union.slt
index ca83e4b652..3a7bd7f881 100644
--- a/datafusion/sqllogictest/test_files/union.slt
+++ b/datafusion/sqllogictest/test_files/union.slt
@@ -1144,6 +1144,20 @@ query TITT
----
ns3 4 t3 m
+# repro for https://github.com/apache/datafusion/issues/26516
+# should not error when only a later union input remains
+query I
+SELECT 1 WHERE false UNION ALL (SELECT 2 LIMIT 1);
+----
+2
+
+query I
+SELECT a FROM (VALUES (1)) t(a) WHERE false
+UNION ALL
+(SELECT b FROM (VALUES (2)) s(b) LIMIT 1);
+----
+2
+
# Config reset
statement ok
RESET datafusion.execution.batch_size;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org