[GitHub] [spark] viirya commented on a change in pull request #33910: [SPARK-36666][SQL] Fix regression in AQEShuffleReadExec

[GitBox <gi...@apache.org>]

viirya commented on a change in pull request #33910:
URL: https://github.com/apache/spark/pull/33910#discussion_r702254989



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala
##########
@@ -82,8 +82,11 @@ case class AQEShuffleReadExec private(
         // `RoundRobinPartitioning` but we don't need to retain the number of partitions.
         case r: RoundRobinPartitioning =>
           r.copy(numPartitions = partitionSpecs.length)
-        case other => throw new IllegalStateException(
-          "Unexpected partitioning for coalesced shuffle read: " + other)
+        case _ =>
+          // Spark plugins may have custom partitioning and may replace this operator
+          // during the postStageOptimization phase, so return UnknownPartitioning here
+          // rather than throw an exception
+          UnknownPartitioning(partitionSpecs.length)

Review comment:
       Hmm, this was asked and answered by @cloud-fan in a comment https://github.com/apache/spark/pull/33541/files#r678751988. So the plugin uses a custom shuffle partitioning?
   
   
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org