[GitHub] [spark] andygrove commented on a change in pull request #33910: [SPARK-36666][SQL] Fix regression in AQEShuffleReadExec

[GitBox <gi...@apache.org>]

andygrove commented on a change in pull request #33910:
URL: https://github.com/apache/spark/pull/33910#discussion_r702285879



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala
##########
@@ -82,8 +82,11 @@ case class AQEShuffleReadExec private(
         // `RoundRobinPartitioning` but we don't need to retain the number of partitions.
         case r: RoundRobinPartitioning =>
           r.copy(numPartitions = partitionSpecs.length)
-        case other => throw new IllegalStateException(
-          "Unexpected partitioning for coalesced shuffle read: " + other)
+        case _ =>
+          // Spark plugins may have custom partitioning and may replace this operator
+          // during the postStageOptimization phase, so return UnknownPartitioning here
+          // rather than throw an exception
+          UnknownPartitioning(partitionSpecs.length)

Review comment:
       We have a GPU version of `HashPartitioning`. When the above code runs, we have already executed a shuffle on the GPU that has `GpuHashPartitioning` and we need to replace `AQEShufflerReader` with a GPU version.
   
   I am going to contribute some tests that have a Spark plugin that uses custom shuffles so that we can better catch this kind of regressions in the future.  I may have to do this as a separate PR but I will see how much work is involved.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org