You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "cetra3 (via GitHub)" <gi...@apache.org> on 2026/08/11 01:32:07 UTC
[PR] Fix self join leaf expressions [datafusion]
cetra3 opened a new pull request, #24242:
URL: https://github.com/apache/datafusion/pull/24242
## Which issue does this PR close?
- Closes https://github.com/apache/datafusion/issues/24241
## Rationale for this change
This is a bugfix for a specific query shape.
## What changes are included in this PR?
Changes the way leaf expressions are extracted when the alias is essentially the same as the column
## Are these changes tested?
Yes, with a test that fails on main currently
## Are there any user-facing changes?
Nope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix self join leaf expressions [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24242:
URL: https://github.com/apache/datafusion/pull/24242#issuecomment-5248158828
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24242?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `78.04878%` with `9 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.98%. Comparing base ([`d5bd10d`](https://app.codecov.io/gh/apache/datafusion/commit/d5bd10d8042772539630e7a00275c7b66cac95da?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6a67b33`](https://app.codecov.io/gh/apache/datafusion/commit/6a67b331614e46d3883def44f91a875bc6d37ec4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24242?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...tafusion/optimizer/src/extract\_leaf\_expressions.rs](https://app.codecov.io/gh/apache/datafusion/pull/24242?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fextract_leaf_expressions.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2V4dHJhY3RfbGVhZl9leHByZXNzaW9ucy5ycw==) | 78.04% | [0 Missing and 9 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24242?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24242 +/- ##
==========================================
- Coverage 80.98% 80.98% -0.01%
==========================================
Files 1106 1106
Lines 383495 383536 +41
Branches 383495 383536 +41
==========================================
+ Hits 310590 310615 +25
- Misses 54572 54575 +3
- Partials 18333 18346 +13
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24242?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix self join leaf expressions [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #24242:
URL: https://github.com/apache/datafusion/pull/24242#discussion_r3775351493
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -833,6 +833,15 @@ fn split_and_push_projection(
for (expr, (qualifier, field)) in proj.expr.iter().zip(original_schema.iter()) {
if let Expr::Alias(alias) = expr
&& alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ && let Expr::Column(inner_col) = alias.expr.as_ref()
+ && inner_col.name == alias.name
+ {
+ // passthrough alias: treat it as one that already exists in the input
+ extractors[0].columns_needed.insert(inner_col.clone());
+ recovery_exprs.push(expr.clone());
+ proj_exprs_captured += 1;
+ } else if let Expr::Alias(alias) = expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
Review Comment:
The logic is quite similar to the 3rd branch `else if let Expr::Column(col) = expr {`
I wonder if we could simplified the logic here to have a more straightforward one instead of many if/else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix self join leaf expressions [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #24242:
URL: https://github.com/apache/datafusion/pull/24242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix self join leaf expressions [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #24242:
URL: https://github.com/apache/datafusion/pull/24242#discussion_r3793837918
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -833,6 +833,15 @@ fn split_and_push_projection(
for (expr, (qualifier, field)) in proj.expr.iter().zip(original_schema.iter()) {
if let Expr::Alias(alias) = expr
&& alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ && let Expr::Column(inner_col) = alias.expr.as_ref()
+ && inner_col.name == alias.name
+ {
+ // passthrough alias: treat it as one that already exists in the input
+ extractors[0].columns_needed.insert(inner_col.clone());
+ recovery_exprs.push(expr.clone());
+ proj_exprs_captured += 1;
+ } else if let Expr::Alias(alias) = expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
Review Comment:
@jayzhan211 I have rewritten this and would appreciate another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix self join leaf expressions [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #24242:
URL: https://github.com/apache/datafusion/pull/24242#discussion_r3779891233
##########
datafusion/optimizer/src/extract_leaf_expressions.rs:
##########
@@ -833,6 +833,15 @@ fn split_and_push_projection(
for (expr, (qualifier, field)) in proj.expr.iter().zip(original_schema.iter()) {
if let Expr::Alias(alias) = expr
&& alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
+ && let Expr::Column(inner_col) = alias.expr.as_ref()
+ && inner_col.name == alias.name
+ {
+ // passthrough alias: treat it as one that already exists in the input
+ extractors[0].columns_needed.insert(inner_col.clone());
+ recovery_exprs.push(expr.clone());
+ proj_exprs_captured += 1;
+ } else if let Expr::Alias(alias) = expr
+ && alias.name.starts_with(EXTRACTED_EXPR_PREFIX)
Review Comment:
This is different in that it's the alias *inner* column that it's being matched on. I guess we could combine the two match arms? But it might make it messier than this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org