You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "naveenp2708 (via GitHub)" <gi...@apache.org> on 2026/04/22 06:27:09 UTC

[PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

naveenp2708 opened a new pull request, #55475:
URL: https://github.com/apache/spark/pull/55475

   ### What changes were proposed in this pull request?
   
   Fix for SPARK-46367. When ProjectExec aliases a column (e.g. `id AS new_id`), `KeyedPartitioning` from `outputPartitioning` still references the old column's ExprId. `EnsureRequirements` cannot match `ClusteredDistribution` on the aliased column and inserts an unnecessary Exchange shuffle.
   
   This fix adds direct ExprId-based remapping of `KeyedPartitioning` expressions through column aliases in `PartitioningPreservingUnaryExecNode`. Two new helpers:
   - `buildExprIdAliasMap`: builds ExprId → Attribute map from alias entries
   - `remapKeyedPartitioning`: substitutes attributes in KeyedPartitioning expressions via the alias map, recursing into transform expressions
   
   Non-aliased attributes absent from the output set cause the partitioning to be dropped, consistent with existing filter logic.
   
   ### Why are the changes needed?
   
   SPJ queries with column aliases followed by aggregation insert unnecessary shuffles, degrading performance. The bug has been present since Spark 3.5.0 and persists on current master after the KeyGroupedPartitioning → KeyedPartitioning refactor.
   
   ### Does this PR introduce any user-facing change?
   
   Yes. SPJ queries with column aliases will avoid unnecessary shuffles for downstream aggregations and dedup operations.
   
   ### How was this patch tested?
   
   Added reproduction test in KeyGroupedPartitioningSuite. All 211 related tests pass.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "naveenp2708 (via GitHub)" <gi...@apache.org>.
naveenp2708 commented on PR #55475:
URL: https://github.com/apache/spark/pull/55475#issuecomment-4294057981

   @peter-toth @szehon-ho This is a fix for SPARK-46367, related to the KeyedPartitioning projection discussion on #54330. The bug has been open since Dec 2023 and confirmed it still reproduces on current master after the KeyedPartitioning refactor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "naveenp2708 (via GitHub)" <gi...@apache.org>.
naveenp2708 commented on PR #55475:
URL: https://github.com/apache/spark/pull/55475#issuecomment-4315189128

   @peter-toth Thank you for the comprehensive fix in #55519! The per-position projection with narrowing support and the isNarrowed flag for groupedSatisfies gating is a much more complete approach. Closing this in favor of yours. Happy to help review or test when ready.- any AI words here like it sound human


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #55475:
URL: https://github.com/apache/spark/pull/55475#issuecomment-4298729336

   @naveenp2708 , this doesn't seem correct to me, the new test should pass without any code change.
   
   Let me show you tomorrow what I was thinking of in https://github.com/apache/spark/pull/54330.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #55475:
URL: https://github.com/apache/spark/pull/55475#issuecomment-4306992893

   @naveenp2708 , I opened a draft PR: https://github.com/apache/spark/pull/55519, but I still need to wrap it up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "naveenp2708 (via GitHub)" <gi...@apache.org>.
naveenp2708 commented on PR #55475:
URL: https://github.com/apache/spark/pull/55475#issuecomment-4299679920

   @peter-toth You're right.I re-ran the test without my fix on clean master and it passes. The existing projectExpression path already handles KeyedPartitioning correctly for this case. Apologies for the false alarm. Looking forward to seeing your approach for the broader projection discussion tomorrow. Thank you for the guidance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec [spark]

Posted by "naveenp2708 (via GitHub)" <gi...@apache.org>.
naveenp2708 closed pull request #55475: [SPARK-46367][SQL] Fix KeyedPartitioning not remapped through column aliases in ProjectExec
URL: https://github.com/apache/spark/pull/55475


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org