You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/25 18:28:37 UTC
[GitHub] [spark] MaxGekk opened a new pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
MaxGekk opened a new pull request #31326:
URL: https://github.com/apache/spark/pull/31326
### What changes were proposed in this pull request?
In the PR, I propose to convert `null` partition values to `"__HIVE_DEFAULT_PARTITION__"` before storing in the `In-Memory` catalog internally. Currently, the `In-Memory` catalog maintains null partitions as `"__HIVE_DEFAULT_PARTITION__"` in file system but as `null` values in memory that could cause some issues like in SPARK-34203.
### Why are the changes needed?
`InMemoryCatalog` stores partitions in the file system in the Hive compatible form, for instance, it converts the `null` partition value to `"__HIVE_DEFAULT_PARTITION__"` but at the same time it keeps null as is internally. That causes an issue demonstrated by the example below:
```
$ ./bin/spark-shell -c spark.sql.catalogImplementation=in-memory
```
```scala
scala> spark.conf.get("spark.sql.catalogImplementation")
res0: String = in-memory
scala> sql("CREATE TABLE tbl (col1 INT, p1 STRING) USING parquet PARTITIONED BY (p1)")
res1: org.apache.spark.sql.DataFrame = []
scala> sql("INSERT OVERWRITE TABLE tbl VALUES (0, null)")
res2: org.apache.spark.sql.DataFrame = []
scala> sql("ALTER TABLE tbl DROP PARTITION (p1 = null)")
org.apache.spark.sql.catalyst.analysis.NoSuchPartitionsException: The following partitions not found in table 'tbl' database 'default':
Map(p1 -> null)
at org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.dropPartitions(InMemoryCatalog.scala:440)
```
### Does this PR introduce _any_ user-facing change?
Yes. After the changes, `ALTER TABLE .. DROP PARTITION` can drop the `null` partition in `In-Memory` catalog:
```scala
scala> spark.table("tbl").show(false)
+----+----+
|col1|p1 |
+----+----+
|0 |null|
+----+----+
scala> sql("ALTER TABLE tbl DROP PARTITION (p1 = null)")
res4: org.apache.spark.sql.DataFrame = []
scala> spark.table("tbl").show(false)
+----+---+
|col1|p1 |
+----+---+
+----+---+
```
### How was this patch tested?
Added new test to `DDLSuite`:
```
$ build/sbt -Phive -Phive-thriftserver "test:testOnly *CatalogedDDLSuite"
```
Authored-by: Max Gekk <ma...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit bfc023501379d28ae2db8708928f4e658ccaa07f)
Signed-off-by: Max Gekk <ma...@gmail.com>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767177037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134457/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563950041
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
Now, the `In-Memory` catalog behaves similarly to Hive external catalog, so, we don't need to distinguish them in tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563961310
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
New test covers both v1 In-Memory and Hive external catalogs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767054298
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39043/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767121642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39043/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767029738
**[Test build #134457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134457/testReport)** for PR 31326 at commit [`9171b0b`](https://github.com/apache/spark/commit/9171b0b47f7e0ca0be655ff40e570c1e5572b2c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563954119
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
shall we do it in master branch as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767276896
thanks, merging to 3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767029738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767121642
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31326:
URL: https://github.com/apache/spark/pull/31326
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767096027
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39043/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767164897
**[Test build #134457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134457/testReport)** for PR 31326 at commit [`9171b0b`](https://github.com/apache/spark/commit/9171b0b47f7e0ca0be655ff40e570c1e5572b2c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563954392
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
A simpler way is to test DROP PARTITION here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767276896
thanks, merging to 3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767177037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134457/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563962939
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
OK, then please forward port the new test to branch-3.1, as I used this smaller change while backporting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767121642
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r564060325
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
Here it is https://github.com/apache/spark/pull/31331
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767121642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39043/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767029738
**[Test build #134457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134457/testReport)** for PR 31326 at commit [`9171b0b`](https://github.com/apache/spark/commit/9171b0b47f7e0ca0be655ff40e570c1e5572b2c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563961310
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
New test covers both v1 In-Memory and Hive external catalogs because it runs as a part of `InMemoryCatalogedDDLSuite` and `HiveCatalogedDDLSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31326:
URL: https://github.com/apache/spark/pull/31326
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563962391
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
In master, we already have common settings in unified tests:
https://github.com/apache/spark/blob/861f8bb5fb82e53a223ae121737fd6d54ab8ba52/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableDropPartitionSuite.scala#L36
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563954119
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
shall we do it in master branch as well?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
A simpler way is to test DROP PARTITION here.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
OK, then please forward port the new test to branch-3.1, as I used this smaller change while backporting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31326:
URL: https://github.com/apache/spark/pull/31326#issuecomment-767029738
**[Test build #134457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134457/testReport)** for PR 31326 at commit [`9171b0b`](https://github.com/apache/spark/commit/9171b0b47f7e0ca0be655ff40e570c1e5572b2c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31326: [SPARK-34203][SQL][3.0] Convert `null` partition values to `__HIVE_DEFAULT_PARTITION__` in v1 `In-Memory` catalog
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31326:
URL: https://github.com/apache/spark/pull/31326#discussion_r563950041
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
Now, the `In-Memory` catalog behaves similarly to Hive external catalog, so, we don't need to distinguish them in tests.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
New test covers both v1 In-Memory and Hive external catalogs.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##########
@@ -1734,9 +1735,8 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
// null partition values
createTablePartition(catalog, Map("a" -> null, "b" -> null), tableIdent)
- val nullPartValue = if (isUsingHiveMetastore) "__HIVE_DEFAULT_PARTITION__" else null
assert(catalog.listPartitions(tableIdent).map(_.spec).toSet ==
- Set(Map("a" -> nullPartValue, "b" -> nullPartValue)))
+ Set(Map("a" -> "__HIVE_DEFAULT_PARTITION__", "b" -> "__HIVE_DEFAULT_PARTITION__")))
Review comment:
In master, we already have common settings in unified tests:
https://github.com/apache/spark/blob/861f8bb5fb82e53a223ae121737fd6d54ab8ba52/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableDropPartitionSuite.scala#L36
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
New test covers both v1 In-Memory and Hive external catalogs because it runs as a part of `InMemoryCatalogedDDLSuite` and `HiveCatalogedDDLSuite`.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3575,15 +3575,6 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
})
}
-
- test("SPARK-33591: null as a partition value") {
- val t = "part_table"
- withTable(t) {
- sql(s"CREATE TABLE $t (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1)")
- sql(s"INSERT INTO TABLE $t PARTITION (p1 = null) SELECT 0")
- checkAnswer(sql(s"SELECT * FROM $t"), Row(0, null))
- }
Review comment:
Here it is https://github.com/apache/spark/pull/31331
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org