You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/16 03:50:34 UTC
[GitHub] [spark] BelodengKlaus opened a new pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
BelodengKlaus opened a new pull request #34012:
URL: https://github.com/apache/spark/pull/34012
### What changes were proposed in this pull request?
Change the uts for parquet compression
### Why are the changes needed?
To check the compression for parquet
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
change uts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921302569
what is "uts"? do you mean unit tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34012:
URL: https://github.com/apache/spark/pull/34012#discussion_r710643374
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
yea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-920554040
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921396686
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47884/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921405883
Thanks for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921381041
+1 for replacing "uts" with "unit tests".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-920673605
@wzhfy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921383678
> +1 for replacing "uts" with "unit tests". it is pretty confusing.
Thanks for the change of title. ^ ^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921383365
> ^^ can you fix PR title and description ut -> unit test?
changed, thanks for the advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921369579
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921380050
^^ can you fix PR title and description ut -> unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921405091
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921370967
**[Test build #143377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143377/testReport)** for PR 34012 at commit [`8236176`](https://github.com/apache/spark/commit/82361766f899b87d52165a80ea5541a4853cb246).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34012:
URL: https://github.com/apache/spark/pull/34012#discussion_r710623550
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921513816
**[Test build #143377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143377/testReport)** for PR 34012 at commit [`8236176`](https://github.com/apache/spark/commit/82361766f899b87d52165a80ea5541a4853cb246).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34012:
URL: https://github.com/apache/spark/pull/34012#discussion_r710621262
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
Parquet uses its own ZSTD codec, see https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921399883
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47884/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921399902
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47884/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on a change in pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on a change in pull request #34012:
URL: https://github.com/apache/spark/pull/34012#discussion_r710645081
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
I have test in my own env with the "LZ4", "ZSTD" and "BROTLI". The "BROTLI" is a hadoop build in plugin, we can not check this compression, and the "ZSTD" and "LZ4" import jar packages to support those compression. So I think we need to check those compression can work well when we upgrade the parquet , zstd and lz4 version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-920554040
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921517977
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921517977
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34012:
URL: https://github.com/apache/spark/pull/34012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BelodengKlaus commented on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
BelodengKlaus commented on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921379379
> what is "uts"? do you mean unit tests?
yes, it means unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34012:
URL: https://github.com/apache/spark/pull/34012#discussion_r710618686
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
`org.apache.hadoop.io.compress.ZStandardCodec` is added at Apache Hadoop 2.9.0+. Does the test pass with Hadoop 2 profile?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompressionCodecPrecedenceSuite.scala
##########
@@ -105,7 +105,7 @@ class ParquetCompressionCodecPrecedenceSuite extends ParquetTest with SharedSpar
test("Create parquet table with compression") {
Seq(true, false).foreach { isPartitioned =>
- Seq("UNCOMPRESSED", "SNAPPY", "GZIP").foreach { compressionCodec =>
+ Seq("UNCOMPRESSED", "SNAPPY", "GZIP", "LZ4", "ZSTD").foreach { compressionCodec =>
Review comment:
cc @viirya too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya edited a comment on pull request #34012: [SPARK-36773] Fixed uts to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
viirya edited a comment on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921381041
+1 for replacing "uts" with "unit tests". it is pretty confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921370967
**[Test build #143377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143377/testReport)** for PR 34012 at commit [`8236176`](https://github.com/apache/spark/commit/82361766f899b87d52165a80ea5541a4853cb246).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34012: [SPARK-36773][SQL][TEST] Fixed unit test to check the compression for parquet
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34012:
URL: https://github.com/apache/spark/pull/34012#issuecomment-921399902
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47884/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org