[jira] [Commented] (SPARK-60203) SPJ with allowJoinKeysSubsetOfPartitionKeys fails in AQE skew-join validation when a non-join partition key comes first

["Ammar Chalifah (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-60203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126490#comment-18126490 ] 

Ammar Chalifah commented on SPARK-60203:
----------------------------------------

Related issue: https://github.com/apache/iceberg/issues/15349

> SPJ with allowJoinKeysSubsetOfPartitionKeys fails in AQE skew-join validation when a non-join partition key comes first
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60203
>                 URL: https://issues.apache.org/jira/browse/SPARK-60203
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 4.1.1, 4.1.2, 4.1.3, 4.0.4
>            Reporter: Ammar Chalifah
>            Priority: Minor
>
> In copy-on-write MERGE operations with Iceberg, for tables that are partitioned on superset of the join keys and join keys doesn't include the first partition key, the operations fails on query planning side.
> For example, a table partitioned on `date, bucket(12, id)`, and MERGE with `id` as the only join key, raised an error:
> {{```}}
> {{{}java.lang.IllegalStateException: Unknown type for int field. Type name: java.lang.String{}}}, 
> ```
> The plan fails at {{{}KeyGroupedPartitioning.createShuffleSpec{}}}, then {{{}KeyGroupedPartitioning.apply{}}}, then {{project}} over {{{}originalPartitionValues{}}}. The scan has already been cut down to the join keys, but {{originalPartitionValues}} were never projected, so Spark reads the full (date, id) row with the join-key types.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org