You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/06 20:56:56 UTC
[GitHub] [spark] MaxGekk opened a new pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
MaxGekk opened a new pull request #34201:
URL: https://github.com/apache/spark/pull/34201
### What changes were proposed in this pull request?
In the PR, I propose to add new test which checks saving/loading of ANSI intervals as columns of a dataframe to/from a table using Hive External catalog and the Parquet datasource.
Since Hive Metastore/Serde doesn't support interval types natively, Spark fallbacks to Spark specific format for schema w/ ANSI intervals. And it outputs the warning:
```
23:35:46.289 WARN org.apache.spark.sql.hive.test.TestHiveExternalCatalog: Could not persist `default`.`tbl_ansi_intervals` in a Hive compatible way. Persisting it into Hive metastore in Spark SQL specific format.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'interval year to month:interval day to second' but 'interval year to month' is found.
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
```
### Why are the changes needed?
To improve test coverage.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
By running new test:
```
$ ./build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *HiveParquetSourceSuite"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937759898
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937477026
Merging to master.
Thank you, @sarutak and @HyukjinKwon for reivew.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937133217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937875637
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143971/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937722217
**[Test build #143971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143971/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937133113
**[Test build #143895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143895/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HiveParquetSourceSuite extends ParquetPartitioningTest with ParquetTest `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937722217
**[Test build #143971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143971/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937759898
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937477026
Merging to master.
Thank you, @sarutak and @HyukjinKwon for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937133217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937123667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #34201:
URL: https://github.com/apache/spark/pull/34201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937873909
**[Test build #143971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143971/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `class HiveParquetSourceSuite extends ParquetPartitioningTest with ParquetTest `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937553608
**[Test build #143947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143947/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937206346
**[Test build #143896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143896/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937553608
**[Test build #143947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143947/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34201:
URL: https://github.com/apache/spark/pull/34201#discussion_r723870451
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala
##########
@@ -382,4 +383,16 @@ class HiveParquetSourceSuite extends ParquetPartitioningTest {
}
}
}
+
+ test("SPARK-36941: Save/load ANSI intervals to Hive Parquet table") {
+ val tableName = "tbl_ansi_intervals"
+ withTable(tableName) {
+ val (ym, dt) = (java.time.Period.ofMonths(10), java.time.Duration.ofDays(1))
+ val df = Seq((ym, dt)).toDF("ym", "dt")
+ df.write.mode(SaveMode.Overwrite).format("parquet").saveAsTable(tableName)
+ withAllParquetReaders {
+ checkAnswer(sql(s"select * from $tableName"), Row(ym, dt))
Review comment:
Other tests in the test suite use this approach. Let's keep this as is for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937123667
**[Test build #143895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143895/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937133217
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143895/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34201:
URL: https://github.com/apache/spark/pull/34201#discussion_r723781527
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala
##########
@@ -382,4 +383,16 @@ class HiveParquetSourceSuite extends ParquetPartitioningTest {
}
}
}
+
+ test("SPARK-36941: Save/load ANSI intervals to Hive Parquet table") {
+ val tableName = "tbl_ansi_intervals"
+ withTable(tableName) {
+ val (ym, dt) = (java.time.Period.ofMonths(10), java.time.Duration.ofDays(1))
+ val df = Seq((ym, dt)).toDF("ym", "dt")
+ df.write.mode(SaveMode.Overwrite).format("parquet").saveAsTable(tableName)
+ withAllParquetReaders {
+ checkAnswer(sql(s"select * from $tableName"), Row(ym, dt))
Review comment:
nit but maybe
```suggestion
checkAnswer(df.table(tableName), Row(ym, dt))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #34201:
URL: https://github.com/apache/spark/pull/34201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937123667
**[Test build #143895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143895/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937204919
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937123667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937739990
**[Test build #143947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143947/testReport)** for PR 34201 at commit [`fa6ee62`](https://github.com/apache/spark/commit/fa6ee62ad359e3f7fa8017cbef835bd0c527df02).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `class HiveParquetSourceSuite extends ParquetPartitioningTest with ParquetTest `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34201:
URL: https://github.com/apache/spark/pull/34201#discussion_r723781527
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala
##########
@@ -382,4 +383,16 @@ class HiveParquetSourceSuite extends ParquetPartitioningTest {
}
}
}
+
+ test("SPARK-36941: Save/load ANSI intervals to Hive Parquet table") {
+ val tableName = "tbl_ansi_intervals"
+ withTable(tableName) {
+ val (ym, dt) = (java.time.Period.ofMonths(10), java.time.Duration.ofDays(1))
+ val df = Seq((ym, dt)).toDF("ym", "dt")
+ df.write.mode(SaveMode.Overwrite).format("parquet").saveAsTable(tableName)
+ withAllParquetReaders {
+ checkAnswer(sql(s"select * from $tableName"), Row(ym, dt))
Review comment:
nit but maybe
```suggestion
checkAnswer(df.table(tableName), Row(ym, dt))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34201:
URL: https://github.com/apache/spark/pull/34201#discussion_r723870451
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala
##########
@@ -382,4 +383,16 @@ class HiveParquetSourceSuite extends ParquetPartitioningTest {
}
}
}
+
+ test("SPARK-36941: Save/load ANSI intervals to Hive Parquet table") {
+ val tableName = "tbl_ansi_intervals"
+ withTable(tableName) {
+ val (ym, dt) = (java.time.Period.ofMonths(10), java.time.Duration.ofDays(1))
+ val df = Seq((ym, dt)).toDF("ym", "dt")
+ df.write.mode(SaveMode.Overwrite).format("parquet").saveAsTable(tableName)
+ withAllParquetReaders {
+ checkAnswer(sql(s"select * from $tableName"), Row(ym, dt))
Review comment:
Other tests in the test suite use this approach. Let's keep this as is for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937149910
jenkins, retest this, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937133217
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143895/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937477026
Merging to master.
Thank you, @sarutak and @HyukjinKwon for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937149910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34201: [SPARK-36941][SQL][TESTS] Check saving/loading of ANSI intervals to Hive Parquet table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34201:
URL: https://github.com/apache/spark/pull/34201#issuecomment-937875637
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143971/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org