You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "ammarchalifah (via GitHub)" <gi...@apache.org> on 2026/10/16 09:43:59 UTC

[PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

ammarchalifah opened a new pull request, #59544:
URL: https://github.com/apache/spark/pull/59544

Fix SPJ originalPartitionValues when join keys are a subset of partition keys
4.0 backport of https://github.com/apache/spark/pull/59410

### What changes were proposed in this pull request?
Project `originalPartitionValues` to the join keys in `KeyGroupedPartitionedScan.getOutputKeyGroupedPartitioning` whenever `spjParams.joinKeyPositions` is set, matching the existing projection of expressions and partitionValues.

Follow-up to [SPARK-58783](https://issues.apache.org/jira/browse/SPARK-58783) (https://github.com/apache/spark/pull/58013). Related: [SPARK-55588](https://issues.apache.org/jira/browse/SPARK-55588). Backport of https://github.com/apache/spark/pull/59410

Targets branch-4.0 only. 4.1 in a separate PR. `master` and `branch-4.2` are unaffected.

Why are the changes needed?
On branch-4.0, SPJ trims the scan partitioning to join keys but leaves full rows in `originalPartitionValues`. A later `KeyGroupedPartitioning.createShuffleSpec` re-projects those rows with the trimmed expressions. When join keys are a non-prefix subset of partition keys (e.g. identity(date), bucket(id), ... joined on id only), a type mismatch could happen and planning fails.

Does this PR introduce any user-facing change?
Yes. Queries that failed at planning time on 4.0.x now run. No change on master or branch-4.2.

How was this patch tested?
Added a test in `KeyGroupedPartitioningSuite`: leading identity(date), join on bucket(32, id) only. Asserts `originalPartitionValues` match each split's raw bucket(32, id) from HasPartitionKey.partitionKey(), SPJ is used, and the join result is correct.

Was this patch authored or co-authored using generative AI tooling?
Yes: Grok 4.7

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.

peter-toth commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6219628121

   Thank you @ammarchalifah. The PR looks good, but I would like to see a green CI run before merging. Can you please recheck if github actions are enabled in your fork and then retrigger the jobs with rebasing all commits on latest `branch-4.0`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "ammarchalifah (via GitHub)" <gi...@apache.org>.

ammarchalifah commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6216276846

   Hi @dongjoon-hyun , I was pretty sure I turned it on, but CI has been flaky


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.

peter-toth commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6262115645

   **Merge Summary:**
   - merged into branch-4.0 https://github.com/apache/spark/commit/d0d3856a3c876f8766a10b5bc82ba43a21dbc969
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "ammarchalifah (via GitHub)" <gi...@apache.org>.

ammarchalifah commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6232029468

   Rebased + pushed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.

uros-b commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6252556588

   Thank you @ammarchalifah! Added reviewers from https://github.com/apache/spark/pull/59410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #59544:
URL: https://github.com/apache/spark/pull/59544#issuecomment-6216215429

   Could you check if your GitHub Action is triggered on your PR, @ammarchalifah ?
   
   cc @peter-toth 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys [spark]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.

peter-toth closed pull request #59544: [SPARK-60203][SQL][4.0] Fix SPJ originalPartitionValues when join keys are a subset of partition keys
URL: https://github.com/apache/spark/pull/59544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org