You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "princejain756 (via GitHub)" <gi...@apache.org> on 2026/10/21 05:24:37 UTC
[PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
princejain756 opened a new pull request, #26551:
URL: https://github.com/apache/datafusion/pull/26551
## Which issue does this PR close?
- Closes #26504.
## Rationale for this change
When a correlated subquery contains an aliased derived table (`SubqueryAlias`) with a correlated predicate inside it and a `Filter` directly above the `SubqueryAlias` whose predicates are all pulled up as join filters (for example `WHERE x.d > 0`), `PullUpCorrelatedExpr::f_up` eliminates the `Filter` and rebuilds `new_plan` equal to `*plan_filter.input` (`SubqueryAlias`).
Previously, `self.correlated_subquery_cols_map.insert(new_plan.clone(), correlated_subquery_cols)` overwrote the entry that `SubqueryAlias` had already recorded for `new_plan`. Because `collect_local_correlated_cols` treats `SubqueryAlias` as a leaf node and does not recurse into its input, any `Projection` or `Aggregate` above the eliminated `Filter` lost the correlated columns from the derived table and failed with `Schema error: No field named __correlated_sq_1.c`.
## What changes are included in this PR?
- Use `.entry(new_plan.clone()).or_default().extend(correlated_subquery_cols)` in `PullUpCorrelatedExpr::f_up` when handling `LogicalPlan::Filter` so existing correlated columns recorded on `new_plan` are preserved when the `Filter` is removed.
- Add optimizer unit tests in `decorrelate_predicate_subquery` and `scalar_subquery_to_join` and sqllogictest coverage in `subquery.slt`.
## Are these changes tested?
Yes, tested via `cargo test -p datafusion-optimizer --lib` and `subquery.slt`.
## Are there any user-facing changes?
Correlated `EXISTS`, `IN`, and scalar subqueries that filter an aliased derived table on an outer column now plan and execute instead of failing with a schema error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #26551:
URL: https://github.com/apache/datafusion/pull/26551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26551:
URL: https://github.com/apache/datafusion/pull/26551#issuecomment-6379350273
Thanks @princejain756 for the quick iterations, and @namanjain24-sudo for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
Posted by "princejain756 (via GitHub)" <gi...@apache.org>.
princejain756 commented on PR #26551:
URL: https://github.com/apache/datafusion/pull/26551#issuecomment-6379085022
Rebased onto `main` and formatted with `cargo fmt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
Posted by "princejain756 (via GitHub)" <gi...@apache.org>.
princejain756 commented on PR #26551:
URL: https://github.com/apache/datafusion/pull/26551#issuecomment-6370453861
Added the `IN` case to `subquery.slt`, thanks for checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(optimizer): preserve correlated columns when removing filter above SubqueryAlias [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26551:
URL: https://github.com/apache/datafusion/pull/26551#issuecomment-6379045254
Once the merge conflict is fixed, this is ready to go. While rebasing, please also run `cargo fmt --all`, since the new test in `scalar_subquery_to_join.rs` isn't formatted yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org