You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ted Yu (Jira)" <ji...@apache.org> on 2022/09/20 20:09:00 UTC

[jira] [Created] (SPARK-40508) Treat unknown partitioning as UnknownPartitioning

Ted Yu created SPARK-40508:
------------------------------

             Summary: Treat unknown partitioning as UnknownPartitioning
                 Key: SPARK-40508
                 URL: https://issues.apache.org/jira/browse/SPARK-40508
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.3.0
            Reporter: Ted Yu


When running spark application against spark 3.3, I see the following :
```
java.lang.IllegalArgumentException: Unsupported data source V2 partitioning type: CustomPartitioning
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:46)
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:34)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
```
The CustomPartitioning works fine with Spark 3.2.1
This PR proposes to relax the code and treat all unknown partitioning the same way as that for UnknownPartitioning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org