You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/28 06:17:51 UTC
[GitHub] [spark] ggershinsky opened a new pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
ggershinsky opened a new pull request #34415:
URL: https://github.com/apache/spark/pull/34415
### What changes were proposed in this pull request?
Bug fix
### Why are the changes needed?
Parquet encryption has a number of modes. One of them is "external key material", which keeps encrypted data keys in a separate file (as opposed to inside the Parquet file). Upon reading, the Spark Parquet connector does not pass the file path, which causes an NPE.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unitest was added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954222193
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953925386
LGTM
cc @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955069246
Merged to 3.2.1/master. Thanks @ggershinsky @sunchao @dbtsai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955060975
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955613389
+1, LGTM. Thank you all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953536308
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954128683
**[Test build #144720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144720/testReport)** for PR 34415 at commit [`d451982`](https://github.com/apache/spark/commit/d451982d683addea07b65fb603e733d9c5cb941b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953902174
jenkins test this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953905006
**[Test build #144720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144720/testReport)** for PR 34415 at commit [`d451982`](https://github.com/apache/spark/commit/d451982d683addea07b65fb603e733d9c5cb941b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954886534
**[Test build #144764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144764/testReport)** for PR 34415 at commit [`044b083`](https://github.com/apache/spark/commit/044b0834fcaba10509b6b47b6dd5a8b529891324).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r738452578
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetFooterReader.java
##########
@@ -47,7 +47,8 @@ public static ParquetMetadata readFooter(Configuration configuration,
private static ParquetMetadata readFooter(HadoopInputFile inputFile,
ParquetMetadataConverter.MetadataFilter filter) throws IOException {
ParquetReadOptions readOptions =
- HadoopReadOptions.builder(inputFile.getConfiguration()).withMetadataFilter(filter).build();
+ HadoopReadOptions.builder(inputFile.getConfiguration(), inputFile.getPath())
+ .withMetadataFilter(filter).build();
Review comment:
super nit: shall we use 2-space indentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954990035
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955059935
**[Test build #144764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144764/testReport)** for PR 34415 at commit [`044b083`](https://github.com/apache/spark/commit/044b0834fcaba10509b6b47b6dd5a8b529891324).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955200286
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954969924
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954885834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953995129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954886534
**[Test build #144764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144764/testReport)** for PR 34415 at commit [`044b083`](https://github.com/apache/spark/commit/044b0834fcaba10509b6b47b6dd5a8b529891324).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r739380976
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala
##########
@@ -64,6 +64,33 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
}
}
+ test("SPARK-37117: Can't read files in Parquet encryption external key material mode") {
Review comment:
Got it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953943535
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953905006
**[Test build #144720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144720/testReport)** for PR 34415 at commit [`d451982`](https://github.com/apache/spark/commit/d451982d683addea07b65fb603e733d9c5cb941b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-955060975
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954130033
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r738551634
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala
##########
@@ -64,6 +64,33 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
}
}
+ test("SPARK-37117: Can't read files in Parquet encryption external key material mode") {
Review comment:
not related, but just curious why this test suite is in sql/hive module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953536308
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953995129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-953995089
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49189/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao closed pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
huaxingao closed pull request #34415:
URL: https://github.com/apache/spark/pull/34415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r738683823
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala
##########
@@ -64,6 +64,33 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
}
}
+ test("SPARK-37117: Can't read files in Parquet encryption external key material mode") {
Review comment:
maybe we should move out hive module to other place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954130033
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ggershinsky commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r739000282
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala
##########
@@ -64,6 +64,33 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
}
}
+ test("SPARK-37117: Can't read files in Parquet encryption external key material mode") {
Review comment:
yep.. In parquet-mr-1.12, PME uses the jackson json library, so we got a suggestion to place the test suite in the Hive module. In the next parquet version (1.13), the PME dependency will change to fasterxml (already merged in parquet-me master), so we'll be able to move the suite to Spark SQL core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ggershinsky commented on a change in pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on a change in pull request #34415:
URL: https://github.com/apache/spark/pull/34415#discussion_r739000282
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetEncryptionSuite.scala
##########
@@ -64,6 +64,33 @@ class ParquetEncryptionSuite extends QueryTest with TestHiveSingleton {
}
}
+ test("SPARK-37117: Can't read files in Parquet encryption external key material mode") {
Review comment:
yep.. In parquet-mr-1.12, PME uses the jackson json library, so we got a suggestion to place the test suite in the Hive module. In the next parquet version (1.13), the PME dependency will change to fasterxml (already merged in parquet-mr master), so we'll be able to move this suite to the Spark SQL core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954222193
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954941824
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34415: [SPARK-37117][SQL] Fix reading encrypted parquet files with external key material
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34415:
URL: https://github.com/apache/spark/pull/34415#issuecomment-954990035
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org