You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2026/08/27 06:45:43 UTC
[PR] [SPARK-59045][SQL] Fix SPJ throw ClassCastException with different data type transforms [spark]
ulysses-you opened a new pull request, #58335:
URL: https://github.com/apache/spark/pull/58335
### What changes were proposed in this pull request?
`KeyedShuffleSpec.reducers` now returns the reduced expression (a `TransformExpression`) alongside
the reducer, and `GroupPartitionsExec.outputPartitioning` reports it instead of the original
partition expressions when reducers are applied.
### Why are the changes needed?
In a storage-partitioned join with compatible transforms whose result types differ (e.g.
`identity(id)` on one side and `bucket(N, id)` on the other), the reducer maps the partition keys to
the other side's value type. `GroupPartitionsExec.outputPartitioning` used to report the original
expressions with the reduced keys, so the two had different data types and computing the key
ordering threw:
```
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added regression tests in `KeyGroupedPartitioningSuite` covering the identity-vs-bucket reducer, the
multi-table reduce (three tables with different transforms), and the subset-join-key case. Each also
asserts the storage-partitioned join introduces no shuffle. The tests fail with the
`ClassCastException` before this patch and pass after.
Ran `KeyGroupedPartitioningSuite`, `EnsureRequirementsSuite`, `GroupPartitionsExecSuite`,
`ProjectedOrderingAndPartitioningSuite`, and `ShuffleSpecSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5498233027
Seems like this causes a test failure on `branch-4.3`:
This is from https://github.com/apache/spark/actions/runs/33502924519/job/99854844877:
```
2026-09-01T14:03:48.8434765Z [0m[[0m[0minfo[0m] [0m[0m[31m- SPARK-59045: compatible transforms reduce data type with subset join keys *** FAILED *** (326 milliseconds)[0m[0m
2026-09-01T14:03:48.8442493Z [0m[[0m[0minfo[0m] [0m[0m[31m List(Exchange hashpartitioning(id#1257917L, 5), ENSURE_REQUIREMENTS, [plan_id=1685013][0m[0m
2026-09-01T14:03:48.8443966Z [0m[[0m[0minfo[0m] [0m[0m[31m +- *(1) Project [id#1257917L, dt#1257918, data#1257919][0m[0m
2026-09-01T14:03:48.8444868Z [0m[[0m[0minfo[0m] [0m[0m[31m +- *(1) Filter isnotnull(id#1257917L)[0m[0m
2026-09-01T14:03:48.8446361Z [0m[[0m[0minfo[0m] [0m[0m[31m +- BatchScan testcat.ns.t1[id#1257917L, dt#1257918, data#1257919] class org.apache.spark.sql.connector.catalog.InMemoryBaseTable$InMemoryBatchScan RuntimeFilters: [][0m[0m
2026-09-01T14:03:48.8447458Z [0m[[0m[0minfo[0m] [0m[0m[31m , Exchange hashpartitioning(id#1257920L, 5), ENSURE_REQUIREMENTS, [plan_id=1685035][0m[0m
2026-09-01T14:03:48.8448392Z [0m[[0m[0minfo[0m] [0m[0m[31m +- *(2) Project [id#1257920L, dt#1257921, data#1257922][0m[0m
2026-09-01T14:03:48.8448965Z [0m[[0m[0minfo[0m] [0m[0m[31m +- *(2) Filter isnotnull(id#1257920L)[0m[0m
2026-09-01T14:03:48.8450163Z [0m[[0m[0minfo[0m] [0m[0m[31m +- BatchScan testcat.ns.t2[id#1257920L, dt#1257921, data#1257922] class org.apache.spark.sql.connector.catalog.InMemoryBaseTable$InMemoryBatchScan RuntimeFilters: [][0m[0m
2026-09-01T14:03:48.8451276Z [0m[[0m[0minfo[0m] [0m[0m[31m ) was not empty storage-partitioned join should not shuffle (KeyGroupedPartitioningSuite.scala:659)[0m[0m
2026-09-01T14:03:48.8452107Z [0m[[0m[0minfo[0m] [0m[0m[31m org.scalatest.exceptions.TestFailedException:[0m[0m
```
cc @HeartSaVioR, @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5498262244
@ulysses-you, do you think you can fix it in a follow-up tailored for `branch-4.3`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5472772027
Updated the PR description (pushed as 265008fc4e8): `reducers` is now `reducersBothWays`, it mentions the canonicalization normalization and the named `IdentityReducer` behind it as well as the new `TransformExpression.withReference`, and the test claims are reworked - only three of the five tests fail on base (identity-vs-bucket, multi-reduce, subset keys); the per-`KeyedPartitioning` retargeting test guards this PR's own re-targeting (as you measured, it regresses together with the multi-reduce test when the use-site re-targeting is dropped), and the canonicalization test arrived with `KeyReducer` in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3900575549
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -4982,11 +5155,14 @@ class KeyGroupedPartitioningSuite
}
}
- test("SPARK-59120: incompatible reduced key types are reported instead of cast") {
- // The first join reduces the identity side onto the year key space. The third table does not
- // reduce at all, so the two key spaces genuinely differ, and reading each side's keys at their
- // own types is what lets `EnsureRequirements` say so. On `master` the merge cast one to the
- // other and threw `ClassCastException`.
+ test("SPARK-59120: a second join with a non-reducing side plans on the reduced keys") {
+ // The first join reduces the identity side onto the year key space and reports the reduced
+ // expression `years(a.ts)`, so the second join's identity side reduces onto it as well and
+ // the whole query plans without a shuffle. Under `keyDataTypes` alone (#58420) the reduced
+ // keys were read at their own types but still reported under the stale `identity(ts)`
+ // expression, and this query raised STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES; before
Review Comment:
Trimmed in 6ece99a8795 - both SPARK-59120 test comments now just describe what they pin; the history lives in #58420's description and tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3891234986
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -59,7 +58,7 @@ case class GroupPartitionsExec(
child: SparkPlan,
@transient joinKeyPositions: Option[Seq[Int]] = None,
@transient expectedPartitionKeys: Option[Seq[(InternalRowComparableWrapper, Int)]] = None,
- @transient reducers: Option[Seq[Option[Reducer[_, _]]]] = None,
+ @transient reducers: Option[Seq[Option[KeyReducer]]] = None,
Review Comment:
Fixed in 265008fc4e8 along both lines you suggested: `doCanonicalize` now normalizes `reducedExpression` against its own references (the stored expression references the other join side's key, which this node's child does not output), and the synthesized identity reducer is a named `IdentityReducer` case class over the re-targeted transform, whose transform is normalized too. Extended the test with both probes - a reduced expression over an attribute the child does not output, and an identity-derived reducer - plus a negative control (structurally different reducers stay unequal) and a mixed `Some`/`None` multi-key case.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -98,6 +98,14 @@ case class TransformExpression(
}
}
+ /**
+ * Re-targets this partition transform expression at `attr`. A partition transform expression
+ * has a single leaf attribute (`KeyedPartitioning.supportsExpressions`), so this replaces it
+ * and the result reports the same transform over `attr`.
+ */
+ def withReference(attr: Attribute): TransformExpression =
Review Comment:
Done in 265008fc4e8 - the scaladoc now says that any field path above the leaf attribute comes from this expression, and that re-targeting is only faithful when the source and the target key expressions have the same path shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3893905879
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -740,6 +741,164 @@ class KeyGroupedPartitioningSuite
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
+ val cols = Array(
+ Column.create("id", LongType),
+ Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+
+ createTable("t2", cols, Array(bucket(4, "id")))
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data FROM testcat.ns.t1 JOIN testcat.ns.t2 ON t1.id = t2.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
+ checkAnswer(df, Seq(Row(1, "a", "x"), Row(2, "b", "y"), Row(3, "c", "z")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: compatible transforms reduce multiple times") {
+ // t1 is partitioned by identity(id) (Long), t2 by bucket(4, id), t3 by bucket(2, id). The
+ // first join reduces t1 to bucket(4, id) (data type changes), and the second join reduces the
+ // result to bucket(2, id). The reduced expression reported by the first join must remain a
+ // ReducibleFunction so the second reduction can be computed.
+ // This test deliberately leaves `V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS` off: both
+ // joins use the whole partition key, and without the config the failure on base is exercised
+ // through the second, independent trigger (`reduceKeys` at the second join) instead of
+ // `createShuffleSpec` -> `toGrouped`.
+ val cols = Array(Column.create("id", LongType), Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ createTable("t2", cols, Array(bucket(4, "id")))
+ createTable("t3", cols, Array(bucket(2, "id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+ sql("INSERT INTO testcat.ns.t3 VALUES (1, 'p'), (2, 'q'), (3, 'r')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data, t3.data FROM testcat.ns.t1 " +
+ "JOIN testcat.ns.t2 ON t1.id = t2.id JOIN testcat.ns.t3 ON t1.id = t3.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
+ checkAnswer(df, Seq(
+ Row(1, "a", "x", "p"), Row(2, "b", "y", "q"), Row(3, "c", "z", "r")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: reduced expression is retargeted per KeyedPartitioning") {
+ // A chained SPJ's output partitioning reports one `KeyedPartitioning` per join side, but the
+ // reduced expression is derived from the single spec that `createKeyedShuffleSpec` picks
+ // (`collectFirst`). Re-targeting it at each `KeyedPartitioning`'s own key attribute keeps the
+ // other sides' partitionings intact - otherwise a GROUP BY on the other side's key no longer
+ // sees a partitioning on it and the query shuffles (0 shuffles on base, 1 after the fix).
Review Comment:
Fixed in c161a21d522 with your wording - the 1 belongs to the ablation, not to the fix.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1382,44 +1422,82 @@ case class KeyedShuffleSpec(
}
/**
- * Return a set of [[Reducer]] for the partition expressions of this shuffle spec,
- * on the partition expressions of another shuffle spec.
- * <p>
- * A [[Reducer]] exists for a partition expression function of this shuffle spec if it is
- * 'reducible' on the corresponding partition expression function of the other shuffle spec.
+ * Compute the reducers for both sides of a join between this shuffle spec and `other`, in a
+ * single pass over the two sides' partition expressions. A pair's reducer lookups
+ * (`TransformExpression.reducers`) are shared between the two directions: the reverse lookup a
+ * direction needs to detect a single-side reduce is exactly the other direction's reducer, so
+ * this materializes each catalog `Reducer` once per direction.
* <p>
- * If a value is returned, there must be one [[Reducer]] per partition expression.
- * A None value in the set indicates that the particular partition expression is not reducible
- * on the corresponding expression on the other shuffle spec.
+ * A [[Reducer]] exists for a partition expression of one side if it is 'reducible' on the
+ * corresponding partition expression of the other side. If a side's value is returned, there
+ * must be one entry per partition expression of that side. A None entry indicates that the
+ * particular partition expression is not reducible on the corresponding expression.
* <p>
- * Returning none also indicates that none of the partition expressions can be reduced on the
- * corresponding expression on the other shuffle spec.
+ * Returning none for a side indicates that none of its partition expressions can be reduced on
+ * the corresponding expression of the other side.
*
* @param other other key-grouped shuffle spec
+ * @return the reducers for this side and for `other` w.r.t. each other
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
- val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ def reducersBothWays(other: KeyedShuffleSpec)
+ : (Option[Seq[Option[KeyReducer]]], Option[Seq[Option[KeyReducer]]]) = {
+ val results: Seq[(Option[KeyReducer], Option[KeyReducer])] =
+ partitioning.expressions.zip(other.partitioning.expressions).map {
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ val thisReducer = e1.reducers(e2)
+ val otherReducer = e2.reducers(e1)
+
+ val thisResult = thisReducer.map { reducer =>
+ if (otherReducer.isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report the target transform (re-targeted at the reporting
+ // partitioning's key attribute by `GroupPartitionsExec`) instead of the un-reduced
+ // `e1`, whose type can differ from the reduced keys. A connector that violates the
+ // contract with a reducer of a different result type fails the reduced-types check in
+ // `EnsureRequirements`, since the other side's keys are typed by the target transform.
+ KeyReducer(reducer, e2)
+ } else {
+ // Both sides reduce: the reduced keys are r1(f1(x)) = r2(f2(x)), which no single
+ // transform describes. Keep reporting the original expression; its type can differ
+ // from the reduced keys, which a downstream shuffle or grouping can then fail on with
+ // a ClassCastException. Known gap, tracked in the follow-up for SPARK-59045.
Review Comment:
Done in c161a21d522 - the gap comment (and the PR description) name SPARK-59121 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5493749106
Now that this is in, one question about the branch reach.
It went to `master`, `branch-4.3` and `branch-4.x`, but not to `branch-4.2`, and the JIRA still lists 5.0.0 as the only affected version.
`branch-4.2` looks affected to me. I read the two sites there: `KeyedShuffleSpec.reducers` still has the identity-vs-transform arm that hands back a bare reducer, and `GroupPartitionsExec.outputPartitioning` still reports the un-reduced expressions. SPARK-59120 was backported to that branch, so the `keyDataTypes` half is already in place there.
Would you like a `[SPARK-59045][SQL][4.2]` backport? I am happy to open one.
I ask because SPARK-59121, the follow-up for the shape where both sides reduce, is a change on top of this one, so it can only reach 4.2 after this does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3900172904
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -98,6 +98,18 @@ case class TransformExpression(
}
}
+ /**
+ * Re-targets this partition transform expression at `attr`. A partition transform expression
+ * has a single leaf attribute (`KeyedPartitioning.supportsExpressions`), so this replaces that
+ * attribute and keeps the rest of the expression: any field path above the leaf comes from this
+ * expression, not from the re-targeted key. Re-targeting is therefore only faithful when the
+ * source and the target key expressions have the same path shape (all producers today use bare
Review Comment:
Removed in a74b22fd7e1 - the scaladoc keeps just the path-shape rule.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1332,6 +1332,46 @@ case class CoalescedHashShuffleSpec(
override def numPartitions: Int = partitions.length
}
+/**
+ * A [[Reducer]] paired with the reduced partition expression it produces.
+ *
+ * When a key-grouped partitioning is reduced onto another partitioning's key space, the original
Review Comment:
Added an example in a74b22fd7e1: an `identity(id)` side joined to a `bucket(4, id)` side reduces with `IdentityReducer(bucket(4, id))`, which maps each raw id value to its `bucket(4, id)` value. The concrete result is up to the connector's bucket algorithm, so the doc does not state one.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -5002,22 +5165,21 @@ class KeyGroupedPartitioningSuite
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
- checkError(
- exception = intercept[SparkException](sql(query).collect()),
- condition = "STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES",
- parameters = Map(
- "leftReducers" -> "[]",
- "leftReducedDataTypes" -> "[\"INT\"]",
- "rightReducers" -> "[]",
- "rightReducedDataTypes" -> "[\"TIMESTAMP\"]"))
+ val df = sql(query)
+ checkAnswer(df, Seq(Row(1, "aa"), Row(2, "bb")))
+ assert(collectAllShuffles(df.queryExecution.executedPlan).isEmpty,
+ "the second join reduces onto the type-correct reported expression")
}
}
}
- test("SPARK-59120: another child is not shuffled onto reducer-rewritten keys") {
- // `canCreatePartitioning` refuses the reduced side as a layout for the second join, so both of
- // that join's sides are shuffled. Without the gate the driver dies while assembling that
- // shuffle's key map, where the stored keys are re-wrapped at the expressions' types.
+ test("SPARK-59120: another child is shuffled onto the type-correct reduced keys") {
+ // This PR reports the reduced side's expression as `years(a.ts)`, which describes the reduced
Review Comment:
Reworded in a74b22fd7e1 - the comment now describes what the test pins and what the base did, without self-reference.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -4982,11 +5142,14 @@ class KeyGroupedPartitioningSuite
}
}
- test("SPARK-59120: incompatible reduced key types are reported instead of cast") {
- // The first join reduces the identity side onto the year key space. The third table does not
- // reduce at all, so the two key spaces genuinely differ, and reading each side's keys at their
- // own types is what lets `EnsureRequirements` say so. On `master` the merge cast one to the
- // other and threw `ClassCastException`.
+ test("SPARK-59120: a second join with a non-reducing side plans on the reduced keys") {
+ // The first join reduces the identity side onto the year key space. This PR reports the
Review Comment:
Reworded in a74b22fd7e1, same as the other test.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1423,44 +1463,82 @@ case class KeyedShuffleSpec(
}
/**
- * Return a set of [[Reducer]] for the partition expressions of this shuffle spec,
- * on the partition expressions of another shuffle spec.
- * <p>
- * A [[Reducer]] exists for a partition expression function of this shuffle spec if it is
- * 'reducible' on the corresponding partition expression function of the other shuffle spec.
+ * Compute the reducers for both sides of a join between this shuffle spec and `other`, in a
+ * single pass over the two sides' partition expressions. A pair's reducer lookups
+ * (`TransformExpression.reducers`) are shared between the two directions: the reverse lookup a
+ * direction needs to detect a single-side reduce is exactly the other direction's reducer, so
+ * this materializes each catalog `Reducer` once per direction.
* <p>
- * If a value is returned, there must be one [[Reducer]] per partition expression.
- * A None value in the set indicates that the particular partition expression is not reducible
- * on the corresponding expression on the other shuffle spec.
+ * A [[Reducer]] exists for a partition expression of one side if it is 'reducible' on the
+ * corresponding partition expression of the other side. If a side's value is returned, there
+ * must be one entry per partition expression of that side. A None entry indicates that the
+ * particular partition expression is not reducible on the corresponding expression.
* <p>
- * Returning none also indicates that none of the partition expressions can be reduced on the
- * corresponding expression on the other shuffle spec.
+ * Returning none for a side indicates that none of its partition expressions can be reduced on
+ * the corresponding expression of the other side.
*
* @param other other key-grouped shuffle spec
+ * @return the reducers for this side and for `other` w.r.t. each other
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
- val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ def reducersBothWays(other: KeyedShuffleSpec)
+ : (Option[Seq[Option[KeyReducer]]], Option[Seq[Option[KeyReducer]]]) = {
+ val results: Seq[(Option[KeyReducer], Option[KeyReducer])] =
+ partitioning.expressions.zip(other.partitioning.expressions).map {
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ val thisReducer = e1.reducers(e2)
+ val otherReducer = e2.reducers(e1)
+
+ val thisResult = thisReducer.map { reducer =>
+ if (otherReducer.isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report the target transform (re-targeted at the reporting
+ // partitioning's key attribute by `GroupPartitionsExec`) instead of the un-reduced
+ // `e1`, whose type can differ from the reduced keys. A connector that violates the
+ // contract with a reducer of a different result type fails the reduced-types check in
+ // `EnsureRequirements`, since the other side's keys are typed by the target transform.
+ KeyReducer(reducer, e2)
Review Comment:
Done in a74b22fd7e1 - the SPARK-56046 same-result-types test now also joins a third table partitioned by `identity(time)`, asserting the answer and no shuffles. Verified by ablation: reporting the stale original expression fails it (the identity side then reduces onto the stale expression, and the shape gate refuses the mis-described reduced layout).
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -740,6 +741,165 @@ class KeyGroupedPartitioningSuite
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
Review Comment:
Reworded in a74b22fd7e1 - it now explains that another join can reduce onto the reported transform, and a reduced layout can serve as another child's shuffle target, only while the expressions describe the keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ throw ClassCastException with different data type transforms [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3871285779
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +70,22 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions (whose data type matches the reduced
+ // partition keys) are reported instead of the original ones.
val partitionKeys = groupedPartitions.map(_._1)
p.transform {
case k: KeyedPartitioning =>
val projectedExpressions = joinKeyPositions.fold(k.expressions)(_.map(k.expressions))
- KeyedPartitioning(projectedExpressions, partitionKeys, isGrouped = isGrouped)
+ val effectiveExpressions = reducers match {
+ case Some(exprs) =>
+ assert(projectedExpressions.length == exprs.length)
+ projectedExpressions.zip(exprs).map {
+ case (expr, Some((_, reduced))) => reduced
Review Comment:
**Finding 2.** `reduced` was derived from one `KeyedShuffleSpec`, but this `transform` applies it to every `KeyedPartitioning` in the child's partitioning, so the others lose their own key attribute.
`createKeyedShuffleSpec` takes the first `KeyedPartitioning` that satisfies the distribution (`collectFirst`, `EnsureRequirements.scala:819`), and the reducers are computed from that one alone. When this node sits on a join - a chained SPJ - the child reports one `KeyedPartitioning` per side, and all of them get that single `reduced` expression, which references only the chosen side's attribute. The `assert` above has the same root: the length is only guaranteed to match for the spec's own `KeyedPartitioning`.
Three tables bucketed 16, 8 and 4 on `id`, ids 0..15, joined on `a.id` and grouped by `b.id`:
- base: `KPs = [bucket(16, id#18L), bucket(8, id#20L)]`, 0 shuffles.
- this commit: `KPs = [bucket(16, id#18L), bucket(16, id#18L)]`, 1 shuffle, because `GROUP BY b.id` no longer sees a partitioning on `b.id`.
Both return the same rows, so this is an avoidable shuffle rather than a wrong answer. Retargeting the reduced expression at each `KeyedPartitioning`'s own key attribute fixes it, and the single-attribute invariant makes that well defined:
```scala
projectedExpressions.zip(exprs).map {
case (expr, Some((_, reduced))) =>
// `reduced` was derived from the spec's `KeyedPartitioning`; re-target it at this one's key so
// that every `KeyedPartitioning` in a collection keeps its own attribute.
val attr = expr.references.head
reduced.transform { case _: AttributeReference => attr }
case (expr, None) => expr
}
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1376,11 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(
+ other: KeyedShuffleSpec): Option[Seq[Option[(Reducer[_, _], TransformExpression)]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map(reducer => (reducer, e1))
Review Comment:
**Finding 3.** This branch covers two different shapes, and keeping `e1` is wrong in both. Not asking you to fix them here - the second one needs a design decision - but they should not be left implicit either.
There are three reducer shapes, and the reported expression has to be judged per shape:
1. **This side is identity, the other is a transform.** The reduced expression exists and the branch below builds it. Handled, and correctly.
2. **Both sides are transforms, only this side reduces.** The reduced expression also exists: `r(f1(x)) = f2(x)` is the `ReducibleFunction` contract, so it is the other side's transform retargeted at this side's child. This branch reports `e1` instead. `SPARK-56046: Reducers with same result types` is this shape - `days` reduces onto `years`, `YearsFunction.reducer` returns null - and the reported expression stays `days(arrive_time)` where `years(arrive_time)` is what the keys hold.
3. **Both sides reduce.** The keys are `r1(f1(x)) = r2(f2(x))`, a space that neither transform describes, so no substitution can be correct here.
Shape 3 still throws the exception this PR fixes. `reduceKeys` types the keys with `reducer.resultType()` while `e1.dataType` is `e1.function.resultType()`, and the suite already ships a pair where they differ: `DaysFunctionWithToYearsReducerWithLongResult` (`DateType`) and `YearsFunctionWithToYearsReducerWithLongResult` (`IntegerType`) both reduce to `LongType`. Take `SPARK-56164: Reducers with different result types to original keys` and add `V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS -> true`:
```
java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.Integer
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getInt(rows.scala:170)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering.compare(Unknown Source)
at org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning.toGrouped(partitioning.scala:553)
at org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning.createShuffleSpec(partitioning.scala:629)
at org.apache.spark.sql.execution.exchange.ValidateRequirements$.validateInternal(ValidateRequirements.scala:60)
```
`e2` would not help: `years` is `IntegerType` and the keys are `LongType`.
The other cost of a stale expression is that a *second* join derives its reducer from a transform the data has already left behind. Three tables bucketed 12, 8 and 6 on `id`, ids 0..11: join 1 reduces both sides with `BucketReducer(4)`, so the keys become `id % 4` while the left side reports `bucket(12, id)`; join 2 compares that stale `bucket(12, id)` with `bucket(6, id)`, gets `BucketReducer(6)`, and `(id % 4) % 6` leaves the keys at `id % 4`, while the right side has `gcd == thisNumBuckets`, gets no reducer, and keeps `id % 6`. The two sides are matched across different key spaces: the query returns 4 of 12 rows, 0 shuffles, no error, and all 12 with `allowCompatibleTransforms=false`.
```scala
val cols = Array(Column.create("id", LongType), Column.create("data", StringType))
createTable("b12", cols, Array(bucket(12, "id")))
createTable("b8", cols, Array(bucket(8, "id")))
createTable("b6", cols, Array(bucket(6, "id")))
val values = (0 until 12).map(i => s"($i, 'v$i')").mkString(", ")
Seq("b12", "b8", "b6").foreach(t => sql(s"INSERT INTO testcat.ns.$t VALUES $values"))
val df = sql(
"""SELECT /*+ MERGE(a, b, c) */ a.id FROM testcat.ns.b12 a
|JOIN testcat.ns.b8 b ON a.id = b.id
|JOIN testcat.ns.b6 c ON a.id = c.id""".stripMargin)
withSQLConf(SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
checkAnswer(df, (0 until 12).map(i => Row(i.toLong))) // returns only 0, 1, 2, 3
}
```
That one is pre-existing, base reports the same expression, so it is not something this PR broke. It is not master-only either: the same query returns 8 of 12 rows on `branch-4.1`, where `KeyGroupedPartitionedScan.getOutputKeyGroupedPartitioning` reports the original expressions next to the reduced common partition values in the same way.
The shape I would suggest, in case it is useful: extend the substitution to shape 2, which is a small step from what you already do for shape 1; and for shape 3 stop trying to express the keys as a transform - carry the reduced key data types on `KeyedPartitioning` instead. `reduceKeys` already computes them one frame up and `GroupPartitionsExec.groupedPartitionsTuple` already holds them as `reducedDataTypes`, they are just dropped, and the reported partitioning re-derives its types from `expressions.map(_.dataType)` (`partitioning.scala:545`). All eight readers go through `expressionDataTypes`, so they would pick the carried types up for free. Such a partitioning then has to refuse a further reduction, since its expressions no longer describe its keys - which is exactly what the 12/8/6 case above needs.
Happy to pick this up as a follow-up if you would rather not carry it - tell me which way you prefer.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -542,6 +542,87 @@ class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase with
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
+ val cols = Array(
+ Column.create("id", LongType),
+ Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+
+ createTable("t2", cols, Array(bucket(4, "id")))
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data FROM testcat.ns.t1 JOIN testcat.ns.t2 ON t1.id = t2.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
+ checkAnswer(df, Seq(Row(1, "a", "x"), Row(2, "b", "y"), Row(3, "c", "z")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: compatible transforms reduce multiple times") {
+ // t1 is partitioned by identity(id) (Long), t2 by bucket(4, id), t3 by bucket(2, id). The
+ // first join reduces t1 to bucket(4, id) (data type changes), and the second join reduces the
+ // result to bucket(2, id). The reduced expression reported by the first join must remain a
+ // ReducibleFunction so the second reduction can be computed.
+ val cols = Array(Column.create("id", LongType), Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ createTable("t2", cols, Array(bucket(4, "id")))
+ createTable("t3", cols, Array(bucket(2, "id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+ sql("INSERT INTO testcat.ns.t3 VALUES (1, 'p'), (2, 'q'), (3, 'r')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data, t3.data FROM testcat.ns.t1 " +
+ "JOIN testcat.ns.t2 ON t1.id = t2.id JOIN testcat.ns.t3 ON t1.id = t3.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
Review Comment:
**Finding 4.** All three new tests enable this config, so all three fail on base at the same place, `createShuffleSpec` -> `toGrouped`. There is a second, independent trigger that none of them reach.
This test does not need the config, the join uses the whole partition key. With it removed the test still fails on base, but through `reduceKeys` at the second join, where the reducer is bound to the stale un-reduced type:
```
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long
at org.apache.spark.sql.catalyst.plans.physical.KeyedShuffleSpec$$anon$1.reduce(partitioning.scala:1394)
at org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning$.reduceKeys(partitioning.scala:725)
at org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning.reduceKeys(partitioning.scala:572)
at org.apache.spark.sql.execution.exchange.EnsureRequirements.checkKeyGroupCompatible(EnsureRequirements.scala:561)
```
and it passes on this commit. Test 1 and test 3 do need the config, test 3 genuinely joins on a subset.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1376,11 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(
+ other: KeyedShuffleSpec): Option[Seq[Option[(Reducer[_, _], TransformExpression)]]] = {
Review Comment:
**Finding 6.** `Option[Seq[Option[(Reducer[_, _], TransformExpression)]]]` now appears in six signatures and forces `_1` at the use sites, including the explain string. A small named type would read better and would document which element is which:
```scala
case class KeyReducer(reducer: Reducer[_, _], reducedExpression: TransformExpression)
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +70,22 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions (whose data type matches the reduced
Review Comment:
**Finding 5.** The parenthetical holds for the identity branch only. For two transforms `reducers` carries `e1`, so the reported data type is the un-reduced one and can differ from the keys (finding 3). Worth naming the branch the claim applies to, since the next reader will lean on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3900293511
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -4982,11 +5155,14 @@ class KeyGroupedPartitioningSuite
}
}
- test("SPARK-59120: incompatible reduced key types are reported instead of cast") {
- // The first join reduces the identity side onto the year key space. The third table does not
- // reduce at all, so the two key spaces genuinely differ, and reading each side's keys at their
- // own types is what lets `EnsureRequirements` say so. On `master` the merge cast one to the
- // other and threw `ClassCastException`.
+ test("SPARK-59120: a second join with a non-reducing side plans on the reduced keys") {
+ // The first join reduces the identity side onto the year key space and reports the reduced
+ // expression `years(a.ts)`, so the second join's identity side reduces onto it as well and
+ // the whole query plans without a shuffle. Under `keyDataTypes` alone (#58420) the reduced
+ // keys were read at their own types but still reported under the stale `identity(ts)`
+ // expression, and this query raised STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES; before
Review Comment:
I think its still too much code history narrated here for a test comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3880262752
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -98,6 +98,14 @@ case class TransformExpression(
}
}
+ /**
+ * Re-targets this partition transform expression at `attr`. A partition transform expression
+ * has a single leaf attribute (`KeyedPartitioning.supportsExpressions`), so this replaces it
+ * and the result reports the same transform over `attr`.
+ */
+ def withReference(attr: Attribute): TransformExpression =
Review Comment:
**Finding 9.** The premise is right — `KeyedPartitioning.supportsExpressions` allows one leaf attribute per transform — but it also allows a `GetStructField` above that attribute, and `transform` replaces only the attribute, so the *path* comes from the source expression rather than from the target.
`GroupPartitionsExec.outputPartitioning` calls this with `expr.references.head`, which is the bare attribute. Re-targeting the other side's `bucket(4, GetStructField(other, 0))` at a struct `s` whose own key expression is `bucket(4, GetStructField(s, 1))` therefore reports field 0 — the source's ordinal read on this side's struct.
I could not reach it from a query, so this is a scaladoc point rather than a live defect: a nested partition reference resolves to an `Alias` over `GetStructField`, and `supportsExpressions`' `isReference` does not unwrap `Alias`, so a nested-field partition transform produces no `KeyedPartitioning` at all today (I measured that for `identity("s.a")`, which reports `UnknownPartitioning`). Worth saying in the doc that the caller has to pass the reference of the *whole* key expression, not just an attribute, so that whoever makes nested keys work does not inherit a silent wrong-field report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ throw ClassCastException with different data type transforms [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5435515106
cc @peter-toth @szehon-ho thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3892655776
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1382,44 +1422,82 @@ case class KeyedShuffleSpec(
}
/**
- * Return a set of [[Reducer]] for the partition expressions of this shuffle spec,
- * on the partition expressions of another shuffle spec.
- * <p>
- * A [[Reducer]] exists for a partition expression function of this shuffle spec if it is
- * 'reducible' on the corresponding partition expression function of the other shuffle spec.
+ * Compute the reducers for both sides of a join between this shuffle spec and `other`, in a
+ * single pass over the two sides' partition expressions. A pair's reducer lookups
+ * (`TransformExpression.reducers`) are shared between the two directions: the reverse lookup a
+ * direction needs to detect a single-side reduce is exactly the other direction's reducer, so
+ * this materializes each catalog `Reducer` once per direction.
* <p>
- * If a value is returned, there must be one [[Reducer]] per partition expression.
- * A None value in the set indicates that the particular partition expression is not reducible
- * on the corresponding expression on the other shuffle spec.
+ * A [[Reducer]] exists for a partition expression of one side if it is 'reducible' on the
+ * corresponding partition expression of the other side. If a side's value is returned, there
+ * must be one entry per partition expression of that side. A None entry indicates that the
+ * particular partition expression is not reducible on the corresponding expression.
* <p>
- * Returning none also indicates that none of the partition expressions can be reduced on the
- * corresponding expression on the other shuffle spec.
+ * Returning none for a side indicates that none of its partition expressions can be reduced on
+ * the corresponding expression of the other side.
*
* @param other other key-grouped shuffle spec
+ * @return the reducers for this side and for `other` w.r.t. each other
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
- val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ def reducersBothWays(other: KeyedShuffleSpec)
+ : (Option[Seq[Option[KeyReducer]]], Option[Seq[Option[KeyReducer]]]) = {
+ val results: Seq[(Option[KeyReducer], Option[KeyReducer])] =
+ partitioning.expressions.zip(other.partitioning.expressions).map {
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ val thisReducer = e1.reducers(e2)
+ val otherReducer = e2.reducers(e1)
+
+ val thisResult = thisReducer.map { reducer =>
+ if (otherReducer.isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report the target transform (re-targeted at the reporting
+ // partitioning's key attribute by `GroupPartitionsExec`) instead of the un-reduced
+ // `e1`, whose type can differ from the reduced keys. A connector that violates the
+ // contract with a reducer of a different result type fails the reduced-types check in
+ // `EnsureRequirements`, since the other side's keys are typed by the target transform.
+ KeyReducer(reducer, e2)
+ } else {
+ // Both sides reduce: the reduced keys are r1(f1(x)) = r2(f2(x)), which no single
+ // transform describes. Keep reporting the original expression; its type can differ
+ // from the reduced keys, which a downstream shuffle or grouping can then fail on with
+ // a ClassCastException. Known gap, tracked in the follow-up for SPARK-59045.
Review Comment:
**Finding 12.** The follow-up has a ticket now, so it can be named here.
SPARK-59121, "Fix wrong results when a storage-partitioned join reduces the partition keys of both sides". It covers this shape plus the silent wrong-results half of it.
```suggestion
// a ClassCastException. Known gap, tracked in SPARK-59121.
```
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -740,6 +741,164 @@ class KeyGroupedPartitioningSuite
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
+ val cols = Array(
+ Column.create("id", LongType),
+ Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+
+ createTable("t2", cols, Array(bucket(4, "id")))
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data FROM testcat.ns.t1 JOIN testcat.ns.t2 ON t1.id = t2.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
+ checkAnswer(df, Seq(Row(1, "a", "x"), Row(2, "b", "y"), Row(3, "c", "z")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: compatible transforms reduce multiple times") {
+ // t1 is partitioned by identity(id) (Long), t2 by bucket(4, id), t3 by bucket(2, id). The
+ // first join reduces t1 to bucket(4, id) (data type changes), and the second join reduces the
+ // result to bucket(2, id). The reduced expression reported by the first join must remain a
+ // ReducibleFunction so the second reduction can be computed.
+ // This test deliberately leaves `V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS` off: both
+ // joins use the whole partition key, and without the config the failure on base is exercised
+ // through the second, independent trigger (`reduceKeys` at the second join) instead of
+ // `createShuffleSpec` -> `toGrouped`.
+ val cols = Array(Column.create("id", LongType), Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ createTable("t2", cols, Array(bucket(4, "id")))
+ createTable("t3", cols, Array(bucket(2, "id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+ sql("INSERT INTO testcat.ns.t3 VALUES (1, 'p'), (2, 'q'), (3, 'r')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data, t3.data FROM testcat.ns.t1 " +
+ "JOIN testcat.ns.t2 ON t1.id = t2.id JOIN testcat.ns.t3 ON t1.id = t3.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
+ checkAnswer(df, Seq(
+ Row(1, "a", "x", "p"), Row(2, "b", "y", "q"), Row(3, "c", "z", "r")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: reduced expression is retargeted per KeyedPartitioning") {
+ // A chained SPJ's output partitioning reports one `KeyedPartitioning` per join side, but the
+ // reduced expression is derived from the single spec that `createKeyedShuffleSpec` picks
+ // (`collectFirst`). Re-targeting it at each `KeyedPartitioning`'s own key attribute keeps the
+ // other sides' partitionings intact - otherwise a GROUP BY on the other side's key no longer
+ // sees a partitioning on it and the query shuffles (0 shuffles on base, 1 after the fix).
Review Comment:
**Finding 11.** The parenthetical is the wrong way round, it reads as if the fix introduces the shuffle.
Measured on `265008fc4e8`: base `af0d5c3abd1` gives 0 shuffles and the test passes, this head gives 0 shuffles and passes, and only replacing `reduced.withReference(expr.references.head)` with `reduced` in `GroupPartitionsExec.outputPartitioning` makes it 1 shuffle. So the 1 belongs to the ablation.
```suggestion
// sees a partitioning on it and the query shuffles (0 shuffles on base and here, 1 if the
// use-site re-targeting is dropped).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3897593522
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1332,6 +1332,46 @@ case class CoalescedHashShuffleSpec(
override def numPartitions: Int = partitions.length
}
+/**
+ * A [[Reducer]] paired with the reduced partition expression it produces.
+ *
+ * When a key-grouped partitioning is reduced onto another partitioning's key space, the original
Review Comment:
optional: quick example to understand this?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:
##########
@@ -98,6 +98,18 @@ case class TransformExpression(
}
}
+ /**
+ * Re-targets this partition transform expression at `attr`. A partition transform expression
+ * has a single leaf attribute (`KeyedPartitioning.supportsExpressions`), so this replaces that
+ * attribute and keeps the rest of the expression: any field path above the leaf comes from this
+ * expression, not from the re-targeted key. Re-targeting is therefore only faithful when the
+ * source and the target key expressions have the same path shape (all producers today use bare
Review Comment:
let's remove the comment about 'all producers today'? doesnt seem necessary to understand this function by itself ?
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -5002,22 +5165,21 @@ class KeyGroupedPartitioningSuite
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
- checkError(
- exception = intercept[SparkException](sql(query).collect()),
- condition = "STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES",
- parameters = Map(
- "leftReducers" -> "[]",
- "leftReducedDataTypes" -> "[\"INT\"]",
- "rightReducers" -> "[]",
- "rightReducedDataTypes" -> "[\"TIMESTAMP\"]"))
+ val df = sql(query)
+ checkAnswer(df, Seq(Row(1, "aa"), Row(2, "bb")))
+ assert(collectAllShuffles(df.queryExecution.executedPlan).isEmpty,
+ "the second join reduces onto the type-correct reported expression")
}
}
}
- test("SPARK-59120: another child is not shuffled onto reducer-rewritten keys") {
- // `canCreatePartitioning` refuses the reduced side as a layout for the second join, so both of
- // that join's sides are shuffled. Without the gate the driver dies while assembling that
- // shuffle's key map, where the stored keys are re-wrapped at the expressions' types.
+ test("SPARK-59120: another child is shuffled onto the type-correct reduced keys") {
+ // This PR reports the reduced side's expression as `years(a.ts)`, which describes the reduced
Review Comment:
(here and other tests), can we not have comments about 'this pr' and just a brief explanation why this test is needed?
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -4982,11 +5142,14 @@ class KeyGroupedPartitioningSuite
}
}
- test("SPARK-59120: incompatible reduced key types are reported instead of cast") {
- // The first join reduces the identity side onto the year key space. The third table does not
- // reduce at all, so the two key spaces genuinely differ, and reading each side's keys at their
- // own types is what lets `EnsureRequirements` say so. On `master` the merge cast one to the
- // other and threw `ClassCastException`.
+ test("SPARK-59120: a second join with a non-reducing side plans on the reduced keys") {
+ // The first join reduces the identity side onto the year key space. This PR reports the
Review Comment:
same, let's remove 'this pr' and just explain what test does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5498253119
Please feel free to revert it from `branch-4.3`, @peter-toth .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5493269155
thank you all !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5502944023
thank you @peter-toth and @dongjoon-hyun , created followup for 4.3 https://github.com/apache/spark/pull/58460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5476857079
#58420 is merged to `master`, `branch-4.x` and `branch-4.3`, so this needs a rebase. Two of its query tests will fail on it, both because this PR makes those shapes work rather than because anything broke:
- `SPARK-59120: incompatible reduced key types are reported instead of cast` - the error is no longer raised, the query returns the right rows with no shuffle. The error path stays covered by `SPARK-56046: Reducers with different result types`, so this one can become a "works now" pin.
- `SPARK-59120: another child is not shuffled onto reducer-rewritten keys` - one shuffle instead of two, onto the now type-correct `years(ts)` layout.
Measured on `265008fc4e8` without #58420, with the rest of the SPJ suites green there. The full comparison is on [the #58420 thread](https://github.com/apache/spark/pull/58420#issuecomment-5476854342).
With those two updated and the three items from my last round answered, this is good to go from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3877553921
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -567,7 +567,8 @@ case class KeyedPartitioning(
* Returns the reduced keys and their data types.
*/
def reduceKeys(
- reducers: Seq[Option[Reducer[_, _]]]): (Seq[DataType], Seq[InternalRowComparableWrapper]) =
+ reducers: Seq[Option[KeyReducer]]):
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3876091918
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -713,17 +714,18 @@ object KeyedPartitioning {
def reduceKeys(
keys: Seq[InternalRowComparableWrapper],
dataTypes: Seq[DataType],
- reducers: Seq[Option[Reducer[_, _]]]): (Seq[DataType], Seq[InternalRowComparableWrapper]) = {
+ reducers: Seq[Option[KeyReducer]]):
+ (Seq[DataType], Seq[InternalRowComparableWrapper]) = {
val reducedDataTypes = dataTypes.zip(reducers).map {
- case (_, Some(reducer: Reducer[Any, Any])) => reducer.resultType()
+ case (_, Some(KeyReducer(reducer: Reducer[Any, Any], _))) => reducer.resultType()
Review Comment:
**[design]** The reduced key type now has two unvalidated sources of truth: `reduceKeys` uses `reducer.resultType()` here, while `GroupPartitionsExec.outputPartitioning` reports `reducedExpression.dataType` (the target function's `resultType()`). `Reducer`'s contract implies they agree, but nothing validates it, so a connector whose `Reducer.resultType()` disagrees with the target transform's type recreates the exact expressions-vs-keys mismatch this PR fixes, hidden inside the new mechanism. Deriving the reduced types from `reducedExpression.dataType`, or asserting agreement when constructing `KeyReducer`, would close the drift by construction.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -59,7 +58,7 @@ case class GroupPartitionsExec(
child: SparkPlan,
@transient joinKeyPositions: Option[Seq[Int]] = None,
@transient expectedPartitionKeys: Option[Seq[(InternalRowComparableWrapper, Int)]] = None,
- @transient reducers: Option[Seq[Option[Reducer[_, _]]]] = None,
+ @transient reducers: Option[Seq[Option[KeyReducer]]] = None,
Review Comment:
**[correctness]** `KeyReducer` embeds an exprId-bearing `TransformExpression` in a non-`Expression` constructor arg. `QueryPlan.doCanonicalize` normalizes exprIds only via `mapExpressions`, which does not recurse into the plain `KeyReducer` case class (`@transient` affects serialization only, not `equals`). Pre-PR, a connector returning value-equal `Reducer` instances on the `ReducibleFunction` path (e.g. the test `BucketReducer(divisor)` case class) allowed two identical SPJ subtrees' `GroupPartitionsExec` nodes to compare equal after canonicalization; post-PR the side-specific `AttributeReference` exprIds inside `reducedExpression` break that equality, so exchange/subquery/stage reuse can silently stop deduplicating those subtrees.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
Review Comment:
**[efficiency]** `e2.reducers(e1)` is invoked per expression pair solely to test `.isEmpty`, materializing a reverse `Reducer` via the catalog function's `reducer()` call. `TransformExpression.reducers` is an uncached `def`, and `EnsureRequirements` already calls `spec.reducers` in both directions, so each direction's catalog lookup now runs twice per join-key pair (4 instead of 2) on every SPJ planning pass, including AQE re-planning. A slow or allocating third-party `ReducibleFunction.reducer()` pays this on a discarded probe; computing both directions once per pair would avoid it.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +69,29 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions are reported instead of the original
+ // ones. For the identity-vs-transform and single-side-transform reducers their data type
+ // matches the reduced partition keys by construction; for the both-sides-reduce shape no
+ // single transform describes the keys (see `KeyedShuffleSpec.reducers`).
val partitionKeys = groupedPartitions.map(_._1)
p.transform {
case k: KeyedPartitioning =>
val projectedExpressions = joinKeyPositions.fold(k.expressions)(_.map(k.expressions))
- KeyedPartitioning(projectedExpressions, partitionKeys, isGrouped = isGrouped)
+ val effectiveExpressions = reducers match {
+ case Some(exprs) =>
+ assert(projectedExpressions.length == exprs.length)
+ projectedExpressions.zip(exprs).map {
+ case (expr, Some(KeyReducer(_, reduced))) =>
+ // `reduced` was derived from the single spec that `createKeyedShuffleSpec`
+ // picked (`collectFirst`); re-target it at this `KeyedPartitioning`'s own key
+ // attribute so that every `KeyedPartitioning` in a collection keeps its own.
+ val attr = expr.references.head
+ reduced.transform { case _: AttributeReference => attr }
Review Comment:
**[reuse]** This is the third copy of the `X.transform { case _: AttributeReference => attr }` retargeting idiom (also at `KeyedShuffleSpec.reducers` in both branches), spread across two modules with no shared helper. This copy must stay exactly consistent with how `reducedExpression` was built in `KeyedShuffleSpec.reducers` - a future change to the rebind rule (e.g. per-position rebinding for multi-reference or nested-field transforms) can silently miss one copy, making the reported `outputPartitioning` stop matching the actual reduced keys. A small helper on `TransformExpression` (returning `TransformExpression`, which also removes the `asInstanceOf` casts) would fit this PR.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -567,7 +567,8 @@ case class KeyedPartitioning(
* Returns the reduced keys and their data types.
*/
def reduceKeys(
- reducers: Seq[Option[Reducer[_, _]]]): (Seq[DataType], Seq[InternalRowComparableWrapper]) =
+ reducers: Seq[Option[KeyReducer]]):
Review Comment:
**[style]** Nit: this signature (and `object KeyedPartitioning.reduceKeys` below) was re-wrapped with the bare return type on its own continuation line, but the one-line forms fit within the 100-char limit after the type shortened (`Seq[Option[KeyReducer]]` is shorter than `Seq[Option[Reducer[_, _]]]`), and this wrapping style is not used elsewhere in the file. Keeping the original one-line form avoids the formatting churn.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report that expression (re-targeted at this side's attribute)
+ // instead of the un-reduced `e1`, whose type can differ from the reduced keys.
+ val thisSideChild = e1.references.head
+ val reducedExpr = e2.transform { case _: AttributeReference => thisSideChild }
+ KeyReducer(reducer, reducedExpr.asInstanceOf[TransformExpression])
+ } else {
+ // Both sides reduce: the reduced keys are r1(f1(x)) = r2(f2(x)), which no single
+ // transform describes. Keep reporting the original expression. Known gap, tracked in
+ // the follow-up for SPARK-59045.
+ KeyReducer(reducer, e1)
Review Comment:
**[correctness]** The both-sides-reduce branch keeps the un-reduced `e1`, so the `ClassCastException` this PR fixes remains reachable in that shape with no runtime fail-safe. The shape is reachable even with in-tree test functions (`BucketFunction`'s gcd reducer reduces both ways; `DaysFunctionWithToYearsReducerWithLongResult` / `YearsFunctionWithToYearsReducerWithLongResult` both reduce to `LongType` while their transforms report `DateType`/`IntegerType`). `EnsureRequirements`' `storagePartitionJoinIncompatibleReducedTypesError` check passes here because the LEFT and RIGHT *reduced* types equal each other, so `GroupPartitionsExec` reports original-typed expressions over reduced-typed partition keys, and a downstream GROUP BY / second join / shuffle deriving an ordering from `expressionDataTypes` throws the same CCE.
Since the follow-up is tracked separately, could this branch fail safe until then, e.g. return `None` (fall back to shuffle) or raise the dedicated error when `reducer.resultType() != e1.dataType`?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report that expression (re-targeted at this side's attribute)
+ // instead of the un-reduced `e1`, whose type can differ from the reduced keys.
+ val thisSideChild = e1.references.head
+ val reducedExpr = e2.transform { case _: AttributeReference => thisSideChild }
Review Comment:
**[simplification]** This creation-time retargeting is dead work: the only structural consumer of `reducedExpression` (`GroupPartitionsExec.outputPartitioning`) unconditionally retargets it again at each `KeyedPartitioning`'s own attribute, and all other reads use only `.reducer` (`reduceKeys`, `displayName` in `QueryExecutionErrors`). Storing raw `e2` and doing the single retarget at the use site is equivalent, removes one tree transform + allocation per expression pair, and stops implying that the attribute chosen here (from whichever spec `collectFirst` picked) is load-bearing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3880264752
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -59,7 +58,7 @@ case class GroupPartitionsExec(
child: SparkPlan,
@transient joinKeyPositions: Option[Seq[Int]] = None,
@transient expectedPartitionKeys: Option[Seq[(InternalRowComparableWrapper, Int)]] = None,
- @transient reducers: Option[Seq[Option[Reducer[_, _]]]] = None,
+ @transient reducers: Option[Seq[Option[KeyReducer]]] = None,
Review Comment:
**Finding 8** (my numbering) — an addition to this, measured on `c01535cc6ed`.
The normalization does not fire in the two shapes this PR fixes. Since the third commit stores the raw target transform, `reducedExpression` is the *other* join side's expression: `t` for the identity-vs-transform reducer, `e2` for the single-side-transform one. `QueryPlan.normalizeExpressions(expr, child.output)` rewrites an attribute only when `child.output` contains its exprId, and this node's child is the reduced side, so the other side's attribute is left untouched. It fires only for the both-sides-reduce shape, where `reducedExpression` is this side's own `e1` — the one shape whose expression is stale anyway.
Measured with two `GroupPartitionsExec`s over `LocalTableScanExec(id#1)` and `LocalTableScanExec(id#2)`, a value-equal `BucketReducer(2)`, and `reducedExpression` built over a third attribute standing in for the other side (`oid#11` / `oid#12`):
```
PROBE other-side attribute in reducedExpression, canonical equal = false
```
The same pair with `reducedExpression` over the node's own attribute — the shape the new test builds — gives `true`.
There is a second, independent cause. The synthesized identity reducer is `new Reducer[Any, Any] {...}`, so `KeyReducer.equals` compares it by reference. Taking the reducer from `reducersBothWays` for an `identity(id)` / `bucket(2, id)` pair, with everything else structurally identical:
```
PROBE identity-derived reducer, canonical equal = false
```
That half is pre-existing — the anonymous reducer was already a constructor field of `GroupPartitionsExec` before this PR — so "with value-equal reducers" in the commit message is accurate. It does mean the identity-vs-transform shape, this PR's headline case, still does not deduplicate.
Fix shape: normalize `reducedExpression` against its own references instead of the child's output — the scaladoc already says the attribute it carries is not load-bearing, so a positional canonical form is enough — and make the synthesized reducer a named case class over the target transform so it compares by value, normalizing that expression too, or the exprId problem you just fixed reappears inside the reducer. Extending the new test with an identity-derived reducer, and with a `reducedExpression` over an attribute the child does not output, would pin both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you closed pull request #58335: [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type
URL: https://github.com/apache/spark/pull/58335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3871588965
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +70,22 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions (whose data type matches the reduced
+ // partition keys) are reported instead of the original ones.
val partitionKeys = groupedPartitions.map(_._1)
p.transform {
case k: KeyedPartitioning =>
val projectedExpressions = joinKeyPositions.fold(k.expressions)(_.map(k.expressions))
- KeyedPartitioning(projectedExpressions, partitionKeys, isGrouped = isGrouped)
+ val effectiveExpressions = reducers match {
+ case Some(exprs) =>
+ assert(projectedExpressions.length == exprs.length)
+ projectedExpressions.zip(exprs).map {
+ case (expr, Some((_, reduced))) => reduced
Review Comment:
Fixed in 2b0ef181bc4. `GroupPartitionsExec.outputPartitioning` now re-targets the reduced expression at each `KeyedPartitioning`'s own key attribute before applying it, so a chained SPJ keeps every side's partitioning on its own attribute. Added the 16/8/4 + GROUP BY middle-side regression test; it asserts 0 shuffles and the full 16-row result.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1376,11 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(
+ other: KeyedShuffleSpec): Option[Seq[Option[(Reducer[_, _], TransformExpression)]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map(reducer => (reducer, e1))
Review Comment:
Thanks for the detailed breakdown. Shape 2 is now handled in 2b0ef181bc4: for a single-side reduce we report the target transform re-targeted at this side's attribute, guarded by `e2.reducers(e1).isEmpty`, so both-sides-reduce positions keep the original expression bit-for-bit (shape 3 unchanged, still a tracked gap).
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -542,6 +542,87 @@ class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase with
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
+ val cols = Array(
+ Column.create("id", LongType),
+ Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+
+ createTable("t2", cols, Array(bucket(4, "id")))
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data FROM testcat.ns.t1 JOIN testcat.ns.t2 ON t1.id = t2.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
+ checkAnswer(df, Seq(Row(1, "a", "x"), Row(2, "b", "y"), Row(3, "c", "z")))
+ assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+ "storage-partitioned join should not shuffle")
+ }
+ }
+
+ test("SPARK-59045: compatible transforms reduce multiple times") {
+ // t1 is partitioned by identity(id) (Long), t2 by bucket(4, id), t3 by bucket(2, id). The
+ // first join reduces t1 to bucket(4, id) (data type changes), and the second join reduces the
+ // result to bucket(2, id). The reduced expression reported by the first join must remain a
+ // ReducibleFunction so the second reduction can be computed.
+ val cols = Array(Column.create("id", LongType), Column.create("data", StringType))
+ createTable("t1", cols, Array(identity("id")))
+ createTable("t2", cols, Array(bucket(4, "id")))
+ createTable("t3", cols, Array(bucket(2, "id")))
+ sql("INSERT INTO testcat.ns.t1 VALUES (1, 'a'), (2, 'b'), (3, 'c')")
+ sql("INSERT INTO testcat.ns.t2 VALUES (1, 'x'), (2, 'y'), (3, 'z')")
+ sql("INSERT INTO testcat.ns.t3 VALUES (1, 'p'), (2, 'q'), (3, 'r')")
+
+ val df = sql(
+ "SELECT t1.id, t1.data, t2.data, t3.data FROM testcat.ns.t1 " +
+ "JOIN testcat.ns.t2 ON t1.id = t2.id JOIN testcat.ns.t3 ON t1.id = t3.id")
+
+ withSQLConf(
+ SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true",
+ SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> "true") {
Review Comment:
Done in 2b0ef181bc4. Dropped `V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS` from the multi-table test (both joins use the whole partition key); it now reaches the `reduceKeys` trigger and fails on base at `KeyedShuffleSpec.reduce`. Added a comment noting the two triggers.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +70,22 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions (whose data type matches the reduced
Review Comment:
Fixed in 2b0ef181bc4. The comment now names the branches: the data-type match holds for the identity-vs-transform and single-side-transform reducers, and the both-sides-reduce shape has no single transform expression (pointed at `KeyedShuffleSpec.reducers`).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1376,11 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(
+ other: KeyedShuffleSpec): Option[Seq[Option[(Reducer[_, _], TransformExpression)]]] = {
Review Comment:
Done in 2b0ef181bc4. Added `case class KeyReducer(reducer, reducedExpression)` next to `KeyedShuffleSpec` and switched all signatures/use sites (including the explain string) to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3871609114
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1376,11 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(
+ other: KeyedShuffleSpec): Option[Seq[Option[(Reducer[_, _], TransformExpression)]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map(reducer => (reducer, e1))
Review Comment:
And for shape 3 — please go ahead and take it as a follow-up as you offered; we are happy to leave the both-sides-reduce shape (and the 12/8/6 stale-expression case) out of this PR. Happy to review your follow-up, and to file a JIRA for it if that is useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5450171308
Let me review this again today and check if the new `keyDataTypes` introduced in https://github.com/apache/spark/pull/58262 can help with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3897613748
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -5002,22 +5165,21 @@ class KeyGroupedPartitioningSuite
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
SQLConf.V2_BUCKETING_ALLOW_COMPATIBLE_TRANSFORMS.key -> "true") {
- checkError(
- exception = intercept[SparkException](sql(query).collect()),
- condition = "STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES",
- parameters = Map(
- "leftReducers" -> "[]",
- "leftReducedDataTypes" -> "[\"INT\"]",
- "rightReducers" -> "[]",
- "rightReducedDataTypes" -> "[\"TIMESTAMP\"]"))
+ val df = sql(query)
+ checkAnswer(df, Seq(Row(1, "aa"), Row(2, "bb")))
+ assert(collectAllShuffles(df.queryExecution.executedPlan).isEmpty,
+ "the second join reduces onto the type-correct reported expression")
}
}
}
- test("SPARK-59120: another child is not shuffled onto reducer-rewritten keys") {
- // `canCreatePartitioning` refuses the reduced side as a layout for the second join, so both of
- // that join's sides are shuffled. Without the gate the driver dies while assembling that
- // shuffle's key map, where the stored keys are re-wrapped at the expressions' types.
+ test("SPARK-59120: another child is shuffled onto the type-correct reduced keys") {
+ // This PR reports the reduced side's expression as `years(a.ts)`, which describes the reduced
Review Comment:
(here and other tests), can we not have comments about 'this pr' and just a brief explanation what this is testing
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -740,6 +741,165 @@ class KeyGroupedPartitioningSuite
}
}
+ test("SPARK-59045: compatible identity and bucket transforms reduce data type") {
+ // `identity(id)` reports a Long partition key while `bucket(4, id)` reports an Integer one.
+ // The identity->bucket reducer maps the Long keys to Integer; the GroupPartitionsExec output
+ // partitioning must report the reduced (Integer) expression, not the original Long identity,
+ // or the key ordering derived from the expressions fails with a ClassCastException.
Review Comment:
After #58420, key ordering reads `keyDataTypes`, so leaving the original expression here no longer causes the `ClassCastException` described above; this test passes on the current base. Could this comment instead explain that accurate expressions are required for downstream reduction and layout reuse?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1423,44 +1463,82 @@ case class KeyedShuffleSpec(
}
/**
- * Return a set of [[Reducer]] for the partition expressions of this shuffle spec,
- * on the partition expressions of another shuffle spec.
- * <p>
- * A [[Reducer]] exists for a partition expression function of this shuffle spec if it is
- * 'reducible' on the corresponding partition expression function of the other shuffle spec.
+ * Compute the reducers for both sides of a join between this shuffle spec and `other`, in a
+ * single pass over the two sides' partition expressions. A pair's reducer lookups
+ * (`TransformExpression.reducers`) are shared between the two directions: the reverse lookup a
+ * direction needs to detect a single-side reduce is exactly the other direction's reducer, so
+ * this materializes each catalog `Reducer` once per direction.
* <p>
- * If a value is returned, there must be one [[Reducer]] per partition expression.
- * A None value in the set indicates that the particular partition expression is not reducible
- * on the corresponding expression on the other shuffle spec.
+ * A [[Reducer]] exists for a partition expression of one side if it is 'reducible' on the
+ * corresponding partition expression of the other side. If a side's value is returned, there
+ * must be one entry per partition expression of that side. A None entry indicates that the
+ * particular partition expression is not reducible on the corresponding expression.
* <p>
- * Returning none also indicates that none of the partition expressions can be reduced on the
- * corresponding expression on the other shuffle spec.
+ * Returning none for a side indicates that none of its partition expressions can be reduced on
+ * the corresponding expression of the other side.
*
* @param other other key-grouped shuffle spec
+ * @return the reducers for this side and for `other` w.r.t. each other
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
- val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ def reducersBothWays(other: KeyedShuffleSpec)
+ : (Option[Seq[Option[KeyReducer]]], Option[Seq[Option[KeyReducer]]]) = {
+ val results: Seq[(Option[KeyReducer], Option[KeyReducer])] =
+ partitioning.expressions.zip(other.partitioning.expressions).map {
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ val thisReducer = e1.reducers(e2)
+ val otherReducer = e2.reducers(e1)
+
+ val thisResult = thisReducer.map { reducer =>
+ if (otherReducer.isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report the target transform (re-targeted at the reporting
+ // partitioning's key attribute by `GroupPartitionsExec`) instead of the un-reduced
+ // `e1`, whose type can differ from the reduced keys. A connector that violates the
+ // contract with a reducer of a different result type fails the reduced-types check in
+ // `EnsureRequirements`, since the other side's keys are typed by the target transform.
+ KeyReducer(reducer, e2)
Review Comment:
The new `KeyReducer(reducer, e2)` behavior is important but not directly pinned. A chained `days(ts) JOIN years(ts) JOIN identity(ts)` test would expose regressions: reporting stale `days(ts)` after the first join causes the identity side to reduce to days while the stored keys are years, potentially returning no rows. Could we extend the existing SPARK-56046 test with this third join and assert both the answer and absence of shuffles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5477484639
Rebased onto 7f12c2da0b4 and pushed as c161a21d522; both SPARK-59120 tests are updated as you predicted - the non-reducing identity side now reduces onto the reported `years(a.ts)` and the query plans with no shuffle, and the shape gate accepts the reduced layout so only the unpartitioned side is shuffled onto it (one shuffle; the second test is renamed accordingly).
I also re-measured the fail-on-base claims against the new base: only the multi-reduce test still fails there (CCE - the second reduce's reducer is derived from the stale expression, which no type fix reaches), while the identity-vs-bucket and subset-key cases pass on it now that #58420 fixed the read side. The description's Tested section reflects that.
For the affected versions (finding 10): SPARK-59045 will be updated to 4.2.0/4.3.0/5.0.0 - the crashes start in 4.2.0 with the `KeyedPartitioning`/`GroupPartitionsExec` refactor. The merge covers master (5.0.0) and branch-4.x (4.3.0, which needs a respin since rc1 is cut); no separate branch-4.2 backport for now since the path is opt-in, but happy to cherry-pick if there is demand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5493264586
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/a76ef5775204880c8fbb08190a3cff83d751b986
- merged into branch-4.x https://github.com/apache/spark/commit/ca915bf1a39f527c6977014b5f215dc48c13c333
- merged into branch-4.3 https://github.com/apache/spark/commit/0d253fef08388647dd5336f6737060bd8cbb8f90
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5494359220
@peter-toth created https://github.com/apache/spark/pull/58451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #58335:
URL: https://github.com/apache/spark/pull/58335#issuecomment-5438976642
Thanks for the review — all feedback addressed in 2b0ef181bc4. Summary:
- **1 (title/description):** narrowed the title to the reducer-changes-key-data-type case and rewrote the description: scope is identity-vs-transform and single-side-transform reducers, and the both-sides-reduce shape is called out as a known gap tracked separately.
- **2:** per-`KeyedPartitioning` re-targeting in `outputPartitioning` (blocking regression fixed, with a chained-SPJ test).
- **3:** shape 2 handled (target transform re-targeted at this side's attribute, guarded by `e2.reducers(e1).isEmpty`); shape 3 intentionally left open.
- **4:** multi-table reduce test now exercises the `reduceKeys` trigger without the subset-partition-keys config.
- **5/6:** comment narrowed to the applicable branches; `KeyReducer` case class replaces the tuple across all signatures.
Tests: `KeyGroupedPartitioningSuite` (108), `EnsureRequirementsSuite` + `GroupPartitionsExecSuite` (45) all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #58335:
URL: https://github.com/apache/spark/pull/58335#discussion_r3877548711
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report that expression (re-targeted at this side's attribute)
+ // instead of the un-reduced `e1`, whose type can differ from the reduced keys.
+ val thisSideChild = e1.references.head
+ val reducedExpr = e2.transform { case _: AttributeReference => thisSideChild }
+ KeyReducer(reducer, reducedExpr.asInstanceOf[TransformExpression])
+ } else {
+ // Both sides reduce: the reduced keys are r1(f1(x)) = r2(f2(x)), which no single
+ // transform describes. Keep reporting the original expression. Known gap, tracked in
+ // the follow-up for SPARK-59045.
+ KeyReducer(reducer, e1)
Review Comment:
Thanks for the detailed analysis. We explored both fail-safe options and decided to keep reporting the un-reduced expression until the follow-up, for two reasons:
1. `return None` does not actually fall back to a shuffle. With no reducer on either side, the reduced-types cross-check in `EnsureRequirements` compares the two sides' *original* types (`DateType` vs `IntegerType` in your example) and throws `storagePartitionJoinIncompatibleReducedTypesError` at planning time. We verified this by running the SPARK-56164 query with a per-pair `None` implementation - it failed with exactly that error. So `None` and raising the dedicated error are indistinguishable here: the query cannot run either way, and the error case is already covered by the existing cross-check without new code.
2. `None` additionally opens a silent-wrong-results hole the dedicated error does not have: if both transforms' original types are equal while the reducers' result type differs (constructible by a third-party `ReducibleFunction` pair), both sides return `None`, the cross-check passes on the equal original types, and the two sides' un-reduced key spaces get merged without any reduction.
Since the reduction itself produces correct key values in this shape (only the reported expression is mis-typed), we kept it and left the gap to the follow-up, which carries the reduced data types on `KeyedPartitioning` and fixes this shape properly.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -59,7 +58,7 @@ case class GroupPartitionsExec(
child: SparkPlan,
@transient joinKeyPositions: Option[Seq[Int]] = None,
@transient expectedPartitionKeys: Option[Seq[(InternalRowComparableWrapper, Int)]] = None,
- @transient reducers: Option[Seq[Option[Reducer[_, _]]]] = None,
+ @transient reducers: Option[Seq[Option[KeyReducer]]] = None,
Review Comment:
Fixed in 943811539a8. `GroupPartitionsExec.doCanonicalize` now normalizes the exprIds inside `reducedExpression` via `QueryPlan.normalizeExpressions` against the child's output (the same pattern `BatchScanExec` uses for `keyGroupedPartitioning`), so structurally identical SPJ subtrees with value-equal reducers compare equal after canonicalization again. Added a test that builds two `GroupPartitionsExec`s with differently-numbered exprIds and value-equal `BucketReducer` instances and asserts their canonical forms are equal.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -713,17 +714,18 @@ object KeyedPartitioning {
def reduceKeys(
keys: Seq[InternalRowComparableWrapper],
dataTypes: Seq[DataType],
- reducers: Seq[Option[Reducer[_, _]]]): (Seq[DataType], Seq[InternalRowComparableWrapper]) = {
+ reducers: Seq[Option[KeyReducer]]):
+ (Seq[DataType], Seq[InternalRowComparableWrapper]) = {
val reducedDataTypes = dataTypes.zip(reducers).map {
- case (_, Some(reducer: Reducer[Any, Any])) => reducer.resultType()
+ case (_, Some(KeyReducer(reducer: Reducer[Any, Any], _))) => reducer.resultType()
Review Comment:
Agreed on the drift risk. We first tried asserting agreement when constructing `KeyReducer`, but the in-tree `DaysToYearsReducerWithDateResult` intentionally violates it (`SPARK-56046: Reducers with different result types` depends on such a reducer reaching the cross-check error), so a hard assert would break that test. Instead the agreement is now closed by construction / by the existing check in 943811539a8: the identity-vs-transform reducer reports the transform itself; the single-side branch reports the target transform, whose type the reduced-types cross-check in `EnsureRequirements` validates against the other side's target-transform-typed keys; and the both-sides-reduce branch reports the original expression (known gap, see the other thread). Comments in `reducersBothWays` spell this out.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
+ // Only this side reduces. The reducer contract is r(f1(x)) = f2(x) where "=" matches
+ // both value and data type, so the reduced keys equal the target transform applied to
+ // this side's child. Report that expression (re-targeted at this side's attribute)
+ // instead of the un-reduced `e1`, whose type can differ from the reduced keys.
+ val thisSideChild = e1.references.head
+ val reducedExpr = e2.transform { case _: AttributeReference => thisSideChild }
Review Comment:
Fixed in 943811539a8. The reducer construction now stores the raw expression, and `GroupPartitionsExec.outputPartitioning` does the single re-targeting at the use site (via the new `TransformExpression.withReference` helper).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -1374,9 +1391,25 @@ case class KeyedShuffleSpec(
*
* @param other other key-grouped shuffle spec
*/
- def reducers(other: KeyedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]] = {
+ def reducers(other: KeyedShuffleSpec): Option[Seq[Option[KeyReducer]]] = {
val results = partitioning.expressions.zip(other.partitioning.expressions).map {
- case (e1: TransformExpression, e2: TransformExpression) => e1.reducers(e2)
+ case (e1: TransformExpression, e2: TransformExpression) =>
+ e1.reducers(e2).map { reducer =>
+ if (e2.reducers(e1).isEmpty) {
Review Comment:
Fixed in 943811539a8. The two directions are now computed in one pass, `KeyedShuffleSpec.reducersBothWays`, so each expression pair's `reducers` lookup runs once per direction: the reverse lookup a direction used only for the single-side probe is exactly the other direction's reducer. `EnsureRequirements` now calls it once instead of calling `reducers` on both specs.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -70,11 +69,29 @@ case class GroupPartitionsExec(
// There can be multiple `KeyedPartitioning`s in an output partitioning of a join, but they
// can only differ in `expressions`; their `partitionKeys` reference is shared (enforced by
// `PartitioningCollection`), so `groupedPartitions` is computed only once.
+ // When reducers are applied, the reduced expressions are reported instead of the original
+ // ones. For the identity-vs-transform and single-side-transform reducers their data type
+ // matches the reduced partition keys by construction; for the both-sides-reduce shape no
+ // single transform describes the keys (see `KeyedShuffleSpec.reducers`).
val partitionKeys = groupedPartitions.map(_._1)
p.transform {
case k: KeyedPartitioning =>
val projectedExpressions = joinKeyPositions.fold(k.expressions)(_.map(k.expressions))
- KeyedPartitioning(projectedExpressions, partitionKeys, isGrouped = isGrouped)
+ val effectiveExpressions = reducers match {
+ case Some(exprs) =>
+ assert(projectedExpressions.length == exprs.length)
+ projectedExpressions.zip(exprs).map {
+ case (expr, Some(KeyReducer(_, reduced))) =>
+ // `reduced` was derived from the single spec that `createKeyedShuffleSpec`
+ // picked (`collectFirst`); re-target it at this `KeyedPartitioning`'s own key
+ // attribute so that every `KeyedPartitioning` in a collection keeps its own.
+ val attr = expr.references.head
+ reduced.transform { case _: AttributeReference => attr }
Review Comment:
Fixed in 943811539a8. Added `TransformExpression.withReference(attr)` in catalyst and switched all the copies (the reducer construction in both directions and the use site) to it, which also removes the `asInstanceOf` casts at the call sites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org