You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "HairstonE (via GitHub)" <gi...@apache.org> on 2026/06/11 17:26:30 UTC
[PR] Fix:22477 any all schema error [datafusion]
HairstonE opened a new pull request, #22915:
URL: https://github.com/apache/datafusion/pull/22915
## Which issue does this PR close?
- Closes #22477.
## Rationale for this change
`= ANY (SELECT ...)` and `<> ALL (SELECT ...)` decorrelate into stacked mark joins. The optimizer was pruning each mark join's right child to zero columns, which dropped its table reference. Without a qualifier, every mark column came out named just `mark`, resulting in the schema error.
## What changes are included in this PR?
Mark joins now keep one column from the right child instead of pruning it down to nothing. The right child holds onto its table reference, the mark now stays qualified.
## Are these changes tested?
A unit test builds three stacked mark joins and checks the plan optimizes without the schema error. An sqllogictest runs `= ANY` and `<> ALL` end-to-end against a small table.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #22915:
URL: https://github.com/apache/datafusion/pull/22915#issuecomment-4909862790
Sorry for the delay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #22915:
URL: https://github.com/apache/datafusion/pull/22915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22915:
URL: https://github.com/apache/datafusion/pull/22915#discussion_r3398337828
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -764,12 +780,12 @@ fn split_join_requirements(
// The mark column is synthetic (produced by the join itself),
// so discard it and route only to the left child.
let (left_indices, _mark) = indices.split_off(left_len);
- (left_indices, RequiredIndices::new())
+ (left_indices, RequiredIndices::new().append(&[0]))
Review Comment:
Looks like this code was added in https://github.com/apache/datafusion/pull/21265 from @buraksenn
@buraksenn do you have some time to help review this proposed change?
I think it would help if we can also document in comments why `[0]` is being appended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22915:
URL: https://github.com/apache/datafusion/pull/22915#discussion_r3398485702
##########
datafusion/sqllogictest/test_files/subquery.slt:
##########
@@ -1660,6 +1598,25 @@ logical_plan
21)----------Projection: column1 AS v
22)------------Values: (Int64(5)), (Int64(NULL))
+# same-table `= ANY` / `<> ALL` must plan without
Review Comment:
without the code change in this PR, this test fails as expected
```shell
cargo test --profile=ci --test sqllogictests
...
Completed 475 test files in 9 seconds External error: 2 errors in file /Users/andrewlamb/Software/datafusion3/datafusion/sqllogictest/test_files/subquery.slt
1. query failed: DataFusion error: Optimizer rule 'optimize_projections' failed
caused by
Schema error: Schema contains duplicate unqualified field name mark
[SQL] select id from set_cmp_self where age = any(select age from set_cmp_self);
at /Users/andrewlamb/Software/datafusion3/datafusion/sqllogictest/test_files/subquery.slt:1606
2. query failed: DataFusion error: Optimizer rule 'optimize_projections' failed
caused by
Schema error: Schema contains duplicate unqualified field name mark
[SQL] select id from set_cmp_self where age <> all(select age from set_cmp_self);
at /Users/andrewlamb/Software/datafusion3/datafusion/sqllogictest/test_files/subquery.slt:1613
```
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -764,12 +780,12 @@ fn split_join_requirements(
// The mark column is synthetic (produced by the join itself),
// so discard it and route only to the left child.
let (left_indices, _mark) = indices.split_off(left_len);
- (left_indices, RequiredIndices::new())
+ (left_indices, RequiredIndices::new().append(&[0]))
Review Comment:
It seems like we only need to add the column in some cases -- why does this code add it unconditionally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "HairstonE (via GitHub)" <gi...@apache.org>.
HairstonE commented on PR #22915:
URL: https://github.com/apache/datafusion/pull/22915#issuecomment-4915413116
No worries. Thank you for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix:22477 any all schema error [datafusion]
Posted by "HairstonE (via GitHub)" <gi...@apache.org>.
HairstonE commented on code in PR #22915:
URL: https://github.com/apache/datafusion/pull/22915#discussion_r3403754341
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -764,12 +780,12 @@ fn split_join_requirements(
// The mark column is synthetic (produced by the join itself),
// so discard it and route only to the left child.
let (left_indices, _mark) = indices.split_off(left_len);
- (left_indices, RequiredIndices::new())
+ (left_indices, RequiredIndices::new().append(&[0]))
Review Comment:
I traced the error there and that change guaranteed the column would be non-empty. Typing that out makes me think I shouldn't assume they are empty? So I should only deal with the empty case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org