You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/29 20:00:36 UTC

[GitHub] [spark] MaxGekk opened a new pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

MaxGekk opened a new pull request #30538:
URL: https://github.com/apache/spark/pull/30538


   ### What changes were proposed in this pull request?
   1. Recognize `null` while parsing partition specs, and put `null` instead of `"null"` as partition values.
   2. For V1 catalog: replace `null` by `__HIVE_DEFAULT_PARTITION__`.
   3. For V2 catalogs: pass `null` AS IS, and let catalog implementations to decide how to handle `null`s as partition values in spec.
   
   ### Why are the changes needed?
   Currently, `null` in partition specs is recognized as the `"null"` string which could lead to incorrect results, for example:
   ```sql
   spark-sql> CREATE TABLE tbl5 (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1);
   spark-sql> INSERT INTO TABLE tbl5 PARTITION (p1 = null) SELECT 0;
   spark-sql> SELECT isnull(p1) FROM tbl5;
   false
   ```
   Even we inserted a row to the partition with the `null` value, the resulted table doesn't contain `null`.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes. After the changes, the example above works as expected:
   ```sql
   spark-sql> CREATE TABLE tbl5 (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1);
   spark-sql> INSERT INTO TABLE tbl5 PARTITION (p1 = null) SELECT 0;
   spark-sql> SELECT isnull(p1) FROM tbl5;
   true
   ```
   
   ### How was this patch tested?
   By running the affected test suites `SQLQuerySuite`, `AlterTablePartitionV2SQLSuite` and `v1/ShowPartitionsSuite`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r552945126



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -169,9 +169,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       > Can we add a util function isInvalidPartitionValue?
   
   1. Why are partition values invalid? They are still valid here
   2. Where else will the function be used. Since this is only the place, wouldn't be better to keep the code embedded here?
   
   > will we hit empty string partition value here?
   
   Empty string is handling earlier. We cannot have it here. For example, `SessionCatalog.createPartitions` -> `requireNonEmptyValueInPartitionSpec` which is called before `externalCatalog.createPartitions` where we convert `null` to `__HIVE_DEFAULT_PARTITION__`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r549718294



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -169,9 +169,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       Can we add a util function `isInvalidPartitionValue`? then the code can be
   ```
   case (partitionColumn, value) if isInvalidPartitionValue(value) =>
     isInvalidPartitionValue(spec2(partitionColumn))
   ```
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735455757






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756656382


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38417/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756438626


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38400/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752087460


   **[Test build #133495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133495/testReport)** for PR 30538 at commit [`af2ec3c`](https://github.com/apache/spark/commit/af2ec3c4d558f7deba9e097902aaf74efd369b1f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756734635


   **[Test build #133828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133828/testReport)** for PR 30538 at commit [`89c1572`](https://github.com/apache/spark/commit/89c15724d5af9b423a14f512c8764bd26eee6c9f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752080546


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30538:
URL: https://github.com/apache/spark/pull/30538


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756420560


   **[Test build #133811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133811/testReport)** for PR 30538 at commit [`71ca35a`](https://github.com/apache/spark/commit/71ca35aa17f454bc4d8d6658235a1147b3329416).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r553177340



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -169,9 +169,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       how about `isNullPartitionValue`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756550915


   ```
   [error] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala:183:19: type mismatch;
   [error]  found   : scala.collection.MapView[String,String]
   [error]  required: org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
   [error]     (which expands to)  scala.collection.immutable.Map[String,String]
   [error]     spec.mapValues(v => if (v == null) DEFAULT_PARTITION_NAME else v)
   [error]                   ^
   ```
   
   @MaxGekk It doesn't compile with scala 2.13


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756639815


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38417/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756451399


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38400/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751876709


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38048/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756672354


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38417/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735483533






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735483090


   **[Test build #131936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131936/testReport)** for PR 30538 at commit [`cbf79f1`](https://github.com/apache/spark/commit/cbf79f1f67d83dbe341ba64575920bbc7dc9f6ee).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r552946827



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala
##########
@@ -144,4 +145,14 @@ trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtil
       checkPartitions(t)
     }
   }
+
+  test("SPARK-33591: null as a partition value") {
+    withNamespaceAndTable("ns", "tbl") { t =>
+      sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
+      sql(s"ALTER TABLE $t ADD PARTITION (p1 = null)")

Review comment:
       > don't we have a check to fail null partition value for ADD PARTITION with hive catalog
   
   For hive catalog, we convert `null` to `DEFAULT_PARTITION_NAME` by `ExternalCatalogUtils.convertNullPartitionValues` from `toMetaStorePartitionSpec`. The last one is invoked in all functions in `HiveExternalCatalog` that makes deal with partition specs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749679784


   **[Test build #133236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133236/testReport)** for PR 30538 at commit [`4e4b6cf`](https://github.com/apache/spark/commit/4e4b6cf2dc1c9dcd0116d504981663bb81f3d4be).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756464869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38400/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756420560


   **[Test build #133811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133811/testReport)** for PR 30538 at commit [`71ca35a`](https://github.com/apache/spark/commit/71ca35aa17f454bc4d8d6658235a1147b3329416).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756413806


   **[Test build #133808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133808/testReport)** for PR 30538 at commit [`17938dc`](https://github.com/apache/spark/commit/17938dcb56f60cbf6bb42e50ad3fe87597124463).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755181910


   **[Test build #133730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133730/testReport)** for PR 30538 at commit [`8e3c69a`](https://github.com/apache/spark/commit/8e3c69afcee7ee87cb2fd657573f5aad76a12150).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756519944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133811/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r534136179



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
##########
@@ -950,9 +950,10 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
   // Hive metastore is not case preserving and the partition columns are always lower cased. We need
   // to lower case the column names in partition specification before calling partition related Hive
   // APIs, to match this behaviour.
-  private def lowerCasePartitionSpec(spec: TablePartitionSpec): TablePartitionSpec = {
+  private def toMetaStorePartitionSpec(spec: TablePartitionSpec): TablePartitionSpec = {
     // scalastyle:off caselocale
-    spec.map { case (k, v) => k.toLowerCase -> v }
+    val lowNames = spec.map { case (k, v) => k.toLowerCase -> v }
+    ExternalCatalogUtils.convertNullPartitionValues(lowNames)

Review comment:
       when do we convert empty string to `__HIVE_DEFAULT_PARTITION__`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755353206


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133730/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756616763


   **[Test build #133828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133828/testReport)** for PR 30538 at commit [`89c1572`](https://github.com/apache/spark/commit/89c15724d5af9b423a14f512c8764bd26eee6c9f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756776436


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756464869


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38400/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756394325


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38397/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756512621


   **[Test build #133811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133811/testReport)** for PR 30538 at commit [`71ca35a`](https://github.com/apache/spark/commit/71ca35aa17f454bc4d8d6658235a1147b3329416).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755181910


   **[Test build #133730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133730/testReport)** for PR 30538 at commit [`8e3c69a`](https://github.com/apache/spark/commit/8e3c69afcee7ee87cb2fd657573f5aad76a12150).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756616763






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751852216


   **[Test build #133459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133459/testReport)** for PR 30538 at commit [`af2ec3c`](https://github.com/apache/spark/commit/af2ec3c4d558f7deba9e097902aaf74efd369b1f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749786058


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133236/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756776436






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749991086


   **[Test build #133273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133273/testReport)** for PR 30538 at commit [`343d15d`](https://github.com/apache/spark/commit/343d15d44c67f2139b9a555dbbaa10f684ddb094).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749679315


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37834/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749988674


   **[Test build #133273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133273/testReport)** for PR 30538 at commit [`343d15d`](https://github.com/apache/spark/commit/343d15d44c67f2139b9a555dbbaa10f684ddb094).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751869288


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38048/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756672354






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755212481


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38318/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r553534314



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -133,6 +133,7 @@ case class InsertIntoHiveTable(
     val numDynamicPartitions = partition.values.count(_.isEmpty)
     val numStaticPartitions = partition.values.count(_.nonEmpty)
     val partitionSpec = partition.map {
+      case (key, Some(null)) => key -> ExternalCatalogUtils.DEFAULT_PARTITION_NAME

Review comment:
       The test:
   ```scala
     test("SPARK-33591: '' as a partition value") {
       val t = "part_table"
       withTable(t) {
         sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
         sql(s"INSERT INTO TABLE $t PARTITION (p1 = '') SELECT 0")
       }
     }
   ```
   fails with:
   ```
   Partition spec is invalid. The spec ([p1=Some()]) contains an empty partition column value
   org.apache.spark.sql.AnalysisException: Partition spec is invalid. The spec ([p1=Some()]) contains an empty partition column value
   	at org.apache.spark.sql.execution.datasources.PreprocessTableInsertion$.org$apache$spark$sql$execution$datasources$PreprocessTableInsertion$$preprocess(rules.scala:412)
   ```
   at https://github.com/apache/spark/blob/d730b6bdaa92f2ca19cc8852ac58035e28d47a4f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L408-L409




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735567069


   @cloud-fan @HyukjinKwon Please, take a look at this PR.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756616763


   **[Test build #133828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133828/testReport)** for PR 30538 at commit [`89c1572`](https://github.com/apache/spark/commit/89c15724d5af9b423a14f512c8764bd26eee6c9f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752115049


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38084/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752115031


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38084/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755237432


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38318/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749679315


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37834/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30538:
URL: https://github.com/apache/spark/pull/30538


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755353206


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133730/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735448724


   **[Test build #131936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131936/testReport)** for PR 30538 at commit [`cbf79f1`](https://github.com/apache/spark/commit/cbf79f1f67d83dbe341ba64575920bbc7dc9f6ee).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r549725116



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala
##########
@@ -144,4 +145,14 @@ trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtil
       checkPartitions(t)
     }
   }
+
+  test("SPARK-33591: null as a partition value") {
+    withNamespaceAndTable("ns", "tbl") { t =>
+      sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
+      sql(s"ALTER TABLE $t ADD PARTITION (p1 = null)")

Review comment:
       This is common test suite for all catalogs. Yes, it works for Hive external catalog too.

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala
##########
@@ -144,4 +145,14 @@ trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtil
       checkPartitions(t)
     }
   }
+
+  test("SPARK-33591: null as a partition value") {
+    withNamespaceAndTable("ns", "tbl") { t =>
+      sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
+      sql(s"ALTER TABLE $t ADD PARTITION (p1 = null)")

Review comment:
       This is the common test suite for all catalogs. Yes, it works for Hive external catalog too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r553172509



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
##########
@@ -541,7 +541,12 @@ class InMemoryCatalog(
 
     listPartitions(db, table, partialSpec).map { partition =>
       partitionColumnNames.map { name =>
-        escapePathName(name) + "=" + escapePathName(partition.spec(name))
+        val partValue = if (partition.spec(name) == null) {
+          "null"

Review comment:
       let's use the hive special name.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756830242


   Here are backports:
   - `branch-3.0`: https://github.com/apache/spark/pull/31095
   - `branch-3.1`: https://github.com/apache/spark/pull/31094


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r549758463



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala
##########
@@ -144,4 +145,14 @@ trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtil
       checkPartitions(t)
     }
   }
+
+  test("SPARK-33591: null as a partition value") {
+    withNamespaceAndTable("ns", "tbl") { t =>
+      sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
+      sql(s"ALTER TABLE $t ADD PARTITION (p1 = null)")

Review comment:
       hmm, don't we have a check to fail null partition value for ADD PARTITION with hive catalog? https://github.com/apache/spark/pull/30538/files#diff-9dd0899e5406230aeff96654432da54f35255f6dc60eecb87264a5c508a8c826R1193




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752115049


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38084/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756379516


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38397/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r549718476



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -169,9 +169,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       will we hit empty string partition value here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751876709


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38048/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752092816


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133495/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749785638


   **[Test build #133236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133236/testReport)** for PR 30538 at commit [`4e4b6cf`](https://github.com/apache/spark/commit/4e4b6cf2dc1c9dcd0116d504981663bb81f3d4be).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-750028563


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r534128038



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -167,9 +167,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       shouldn't null be converted to `DEFAULT_PARTITION_NAME` before entering the v1 catalog?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735483533






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-750039767


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749679784


   **[Test build #133236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133236/testReport)** for PR 30538 at commit [`4e4b6cf`](https://github.com/apache/spark/commit/4e4b6cf2dc1c9dcd0116d504981663bb81f3d4be).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r553177619



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -133,6 +133,7 @@ case class InsertIntoHiveTable(
     val numDynamicPartitions = partition.values.count(_.isEmpty)
     val numStaticPartitions = partition.values.count(_.nonEmpty)
     val partitionSpec = partition.map {
+      case (key, Some(null)) => key -> ExternalCatalogUtils.DEFAULT_PARTITION_NAME

Review comment:
       how about empty string?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756419444






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756616763


   **[Test build #133828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133828/testReport)** for PR 30538 at commit [`89c1572`](https://github.com/apache/spark/commit/89c15724d5af9b423a14f512c8764bd26eee6c9f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749991113


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133273/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756357456


   **[Test build #133808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133808/testReport)** for PR 30538 at commit [`17938dc`](https://github.com/apache/spark/commit/17938dcb56f60cbf6bb42e50ad3fe87597124463).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756830242


   Here are backports:
   - `branch-3.0`: https://github.com/apache/spark/pull/31095
   - `branch-3.1`: https://github.com/apache/spark/pull/31094


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755237422


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38318/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752087460


   **[Test build #133495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133495/testReport)** for PR 30538 at commit [`af2ec3c`](https://github.com/apache/spark/commit/af2ec3c4d558f7deba9e097902aaf74efd369b1f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756357456


   **[Test build #133808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133808/testReport)** for PR 30538 at commit [`17938dc`](https://github.com/apache/spark/commit/17938dcb56f60cbf6bb42e50ad3fe87597124463).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756672354


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38417/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735455757






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-750039767


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-735448724


   **[Test build #131936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131936/testReport)** for PR 30538 at commit [`cbf79f1`](https://github.com/apache/spark/commit/cbf79f1f67d83dbe341ba64575920bbc7dc9f6ee).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749988674


   **[Test build #133273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133273/testReport)** for PR 30538 at commit [`343d15d`](https://github.com/apache/spark/commit/343d15d44c67f2139b9a555dbbaa10f684ddb094).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755237432


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38318/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752103176


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38084/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749786058


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133236/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756749901


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133828/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756672354






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756419444






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751946631


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133459/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752092801


   **[Test build #133495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133495/testReport)** for PR 30538 at commit [`af2ec3c`](https://github.com/apache/spark/commit/af2ec3c4d558f7deba9e097902aaf74efd369b1f).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r534128288



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##########
@@ -1141,7 +1141,7 @@ class SessionCatalog(
    */
   private def requireNonEmptyValueInPartitionSpec(specs: Seq[TablePartitionSpec]): Unit = {
     specs.foreach { s =>
-      if (s.values.exists(_.isEmpty)) {
+      if (s.values.exists(v => v != null && v.isEmpty)) {

Review comment:
       ditto




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r552945126



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -169,9 +169,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       > Can we add a util function isInvalidPartitionValue?
   
   1. Why are partition values invalid? They are still valid here
   2. Where else will the function be used. Since this is only the place, wouldn't be better to keep the code embedded here?
   
   > will we hit empty string partition value here?
   
   Empty string is handling earlier. We cannot have it here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756519944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133811/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751861913


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38048/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-755329914


   **[Test build #133730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133730/testReport)** for PR 30538 at commit [`8e3c69a`](https://github.com/apache/spark/commit/8e3c69afcee7ee87cb2fd657573f5aad76a12150).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-749991113


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133273/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-752092816


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133495/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r534128038



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
##########
@@ -167,9 +167,15 @@ object ExternalCatalogUtils {
       spec1: TablePartitionSpec,
       spec2: TablePartitionSpec): Boolean = {
     spec1.forall {
+      case (partitionColumn, null | DEFAULT_PARTITION_NAME) =>

Review comment:
       shouldn't null be converted to `DEFAULT_PARTITION_NAME` before entering the v1 catalog?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756776927


   @MaxGekk can you send backport PRs for 3.1/3.0? thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-751946631


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133459/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30538:
URL: https://github.com/apache/spark/pull/30538#discussion_r549719974



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala
##########
@@ -144,4 +145,14 @@ trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtil
       checkPartitions(t)
     }
   }
+
+  test("SPARK-33591: null as a partition value") {
+    withNamespaceAndTable("ns", "tbl") { t =>
+      sql(s"CREATE TABLE $t (col1 INT, p1 STRING) $defaultUsing PARTITIONED BY (p1)")
+      sql(s"ALTER TABLE $t ADD PARTITION (p1 = null)")

Review comment:
       Does it work for hive catalog?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-750010496


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30538: [SPARK-33591][SQL] Recognize `null` in partition spec values

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30538:
URL: https://github.com/apache/spark/pull/30538#issuecomment-756749901


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133828/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org