You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/08/22 09:50:54 UTC

[PR] fix: scan subqueries when advancing the extracted-alias generator [datafusion]

jayzhan211 opened a new pull request, #24574:
URL: https://github.com/apache/datafusion/pull/24574

   ## Rationale for this change
   
   `ExtractLeafExpressions` rewrites `MoveTowardsLeafNodes` expressions into
   projections aliased `__datafusion_extracted_N`. That prefix is reserved for the
   optimizer, but a user query can still contain it, so before rewriting, the rule
   scans the plan for existing `__datafusion_extracted_N` aliases and advances the
   shared `AliasGenerator` past them to avoid handing out a name that is already
   taken.
   
   That scan used `LogicalPlan::apply`, which does **not** descend into subquery
   plans nested inside expressions — while the rewrite itself uses
   `transform_down_with_subqueries`, which does. So a user alias living inside a
   subquery was invisible to the guard, and extraction could generate that exact
   same alias inside that same subquery. The extraction projection also passes
   through all of its input's columns, so the duplicate name surfaces as an
   ambiguous column reference (planning error) or, worse, silently binds to the
   wrong column.
   
   Concretely, given a subquery containing `... AS __datafusion_extracted_7`, the
   generator still handed out `__datafusion_extracted_1` and counted up from there,
   eventually colliding.
   
   ## What changes are included in this PR?
   
   In `advance_generator_past_existing`:
   
   - `plan.apply` → `plan.apply_with_subqueries`, so the guard covers the same tree
     the rewrite walks. This is the fix.
   - `plan.expressions().iter().try_for_each(...)` → `plan.apply_expressions(...)`.
     `LogicalPlan::expressions()` deep-clones every expression of every node, and
     this scan runs over the whole plan on every invocation of the rule purely to
     *inspect* aliases. `apply_expressions` borrows instead. This also lets the
     nested closure return its `Result<TreeNodeRecursion>` directly, dropping the
     `Ok::<(), DataFusionError>` turbofish.
   
   No behavior change beyond the collision fix; no public API change.
   
   ## Are these changes tested?
   
   Yes — new unit test `test_advance_generator_past_alias_in_subquery`. It builds a
   plan whose `IN (SELECT ...)` subquery contains a user-written
   `AS __datafusion_extracted_7` and asserts the next generated alias is
   `__datafusion_extracted_8`.
   
   The test was confirmed to fail on the pre-fix code
   (`left: "__datafusion_extracted_1"`, `right: "__datafusion_extracted_8"`) and to
   pass after. No existing test covered `advance_generator_past_existing` at all.
   
   Full `cargo test -p datafusion-optimizer` passes (763 + 26 lib/integration tests,
   5 doctests), plus `cargo fmt --all` and
   `cargo clippy --all-targets --all-features -- -D warnings`.
   
   ## Are there any user-facing changes?
   
   No API changes. Queries that use the reserved `__datafusion_extracted_` prefix
   inside a subquery no longer risk an ambiguous-column error or an incorrect
   column binding when leaf expression pushdown is enabled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: scan subqueries when advancing the extracted-alias generator [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24574:
URL: https://github.com/apache/datafusion/pull/24574#discussion_r3837979808


##########
datafusion/sqllogictest/test_files/projection_pushdown.slt:
##########
@@ -2183,3 +2183,74 @@ physical_plan
 # Reset the config changed above (the SLT runner expects target_partitions = 4).
 statement ok
 SET datafusion.execution.target_partitions = 4;
+
+#####################
+# Section: extraction aliases inside a subquery advance the alias generator
+#
+# Regression test for `advance_generator_past_existing` in
+# `extract_leaf_expressions.rs`.
+#
+# `ExtractLeafExpressions` names the columns it extracts
+# `__datafusion_extracted_N`, handing out N from a shared `AliasGenerator`. That
+# prefix is reserved for the optimizer, but nothing stops a user from writing it,
+# so before extracting the rule scans the plan for existing
+# `__datafusion_extracted_N` aliases and bumps the generator past the highest one.
+#
+# The bug: that scan used `apply`, which walks plan nodes but does *not* descend
+# into subquery plans held inside expressions, while the extraction itself uses
+# `transform_down_with_subqueries` and *does* rewrite inside subqueries. So an
+# alias living only inside a subquery was invisible to the scan, and extraction
+# then minted the very same name next to it.
+#
+# Each ingredient below is load-bearing:
+#
+# * The `IN (<subquery>)` sits in the SELECT list, not in a WHERE clause, so
+#   `decorrelate_predicate_subquery` (which runs earlier) leaves it alone and it
+#   is still a subquery expression by the time extraction runs. A subquery in
+#   WHERE would be flattened into the main plan, where the old scan could see it.
+# * The alias inside the subquery is literally `__datafusion_extracted_1`. Rename
+#   it to anything outside the reserved prefix and there is nothing to collide
+#   with -- the query then passes with or without the fix and guards nothing.
+# * The inner `WHERE s['value'] > 120` is what forces extraction to *generate* an
+#   alias inside that same subquery. Without a leaf expression there, the
+#   generator is never called where the collision would happen.
+#
+# Without the fix, extraction reuses `__datafusion_extracted_1` and planning
+# aborts with: Optimizer rule 'push_down_leaf_projections' failed Schema error:
+# Schema contains duplicate unqualified field name __datafusion_extracted_1.
+# With the fix, the generator starts at 2, as the plan below shows.
+#
+# This is `EXPLAIN` under `logical_plan_only` rather than an executed query
+# because a surviving `InSubquery` expression has no physical plan; the logical
+# plan is both the observable result and exactly what regressed.
+#####################
+
+statement ok
+set datafusion.explain.logical_plan_only = true;
+
+query TT
+EXPLAIN
+SELECT
+    id,
+    id IN (
+        SELECT id
+        FROM (
+            SELECT id, s['label'] AS __datafusion_extracted_1

Review Comment:
   I assume there's some way to hit this behavior without literally naming a column `__datafusion_extracted_1`. But since the literal use of `__datafusion_extracted_1` causes an error (and it shouldn't) this is a valid regression test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: scan subqueries when advancing the extracted-alias generator [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24574:
URL: https://github.com/apache/datafusion/pull/24574


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: scan subqueries when advancing the extracted-alias generator [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #24574:
URL: https://github.com/apache/datafusion/pull/24574#issuecomment-5383809671

   I add slt test!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: scan subqueries when advancing the extracted-alias generator [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24574:
URL: https://github.com/apache/datafusion/pull/24574#issuecomment-5379711068

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24574?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `63.15789%` with `7 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 81.36%. Comparing base ([`32badcb`](https://app.codecov.io/gh/apache/datafusion/commit/32badcb7d9e1eaf0980661ada31630a450b14075?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b3e2a30`](https://app.codecov.io/gh/apache/datafusion/commit/b3e2a307cee260a46aedf073aca8b176d739a502?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 2 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24574?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...tafusion/optimizer/src/extract\_leaf\_expressions.rs](https://app.codecov.io/gh/apache/datafusion/pull/24574?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fextract_leaf_expressions.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2V4dHJhY3RfbGVhZl9leHByZXNzaW9ucy5ycw==) | 63.15% | [1 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24574?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #24574      +/-   ##
   ==========================================
   - Coverage   81.37%   81.36%   -0.01%     
   ==========================================
     Files        1116     1116              
     Lines      397509   397563      +54     
     Branches   397509   397563      +54     
   ==========================================
   + Hits       323461   323495      +34     
   - Misses      55110    55119       +9     
   - Partials    18938    18949      +11     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24574?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org