You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/07 16:01:08 UTC
[GitHub] [spark] LuciferYang opened a new pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
LuciferYang opened a new pull request #30652:
URL: https://github.com/apache/spark/pull/30652
### What changes were proposed in this pull request?
As described in SPARK-33673, some test suites in `ParquetV2SchemaPruningSuite` will failed when set `parquet.version` to 1.11.1 because Parquet will return empty results for non-existent column since PARQUET-1765.
This pr change to use `dataSchema` instead of `schema` to build `pushedParquetFilters` in `ParquetScanBuilder` to avoid push down partition filters to `ParquetScan` for `DataSourceV2`
### Why are the changes needed?
Prepare for upgrade using Parquet 1.11.1.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
**Manual test**
```
mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.datasources.parquet.ParquetV2SchemaPruningSuite -Dparquet.version=1.11.1 test -pl sql/core -am
```
**Before**
**After**
```
Run completed in 3 minutes, 46 seconds.
Total number of tests run: 134
Suites: completed 2, aborted 0
Tests: succeeded 134, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744322797
Thanks for your review @wangyum @viirya @cloud-fan @gengliangwang ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744362041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132749/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740593068
Local test `org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite` all passed
```
Discovery starting.
Discovery completed in 906 milliseconds.
Run starting. Expected test count is: 3
HiveThriftHttpServerSuite:
20:03:14.455 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
20:03:14.455 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.transport.mode is deprecated.
20:03:14.455 WARN org.apache.hive.jdbc.Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
20:03:14.456 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
20:03:14.456 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.thrift.http.path is deprecated.
20:03:14.460 WARN org.apache.hive.jdbc.Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
20:03:38.415 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
- JDBC query execution
20:03:38.417 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.transport.mode is deprecated.
20:03:38.417 WARN org.apache.hive.jdbc.Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
20:03:38.417 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
20:03:38.417 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.thrift.http.path is deprecated.
20:03:38.417 WARN org.apache.hive.jdbc.Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
- Checks Hive version
20:03:38.901 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
20:03:38.901 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.transport.mode is deprecated.
20:03:38.901 WARN org.apache.hive.jdbc.Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
20:03:38.901 WARN org.apache.hive.jdbc.Utils: ***** JDBC param deprecation *****
20:03:38.901 WARN org.apache.hive.jdbc.Utils: The use of hive.server2.thrift.http.path is deprecated.
20:03:38.901 WARN org.apache.hive.jdbc.Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
- SPARK-24829 Checks cast as float
Run completed in 52 seconds, 636 milliseconds.
Total number of tests run: 3
Suites: completed 2, aborted 0
Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740012690
cc @wangyum
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r537996738
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
This is a good question, but it seems that the filter pushed down in DataSource V1 does not contain the filter related to partition columns too.
The `dataFilters` use to construct `FileSourceScanExec` and pass to `ParquetFileFormat.buildReaderWithPartitionValues` to build `pushed` filters also filtered out partition filters, am I right?
https://github.com/apache/spark/blob/e4d1c10760800563d2a30410b46e5b0cd2671c4d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L184-L193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740374231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37002/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740033272
**[Test build #132381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132381/testReport)** for PR 30652 at commit [`be6cfc8`](https://github.com/apache/spark/commit/be6cfc8db6e3724da865bd80e06b71bdaa81abc2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740084247
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36981/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741580084
**[Test build #132456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132456/testReport)** for PR 30652 at commit [`b9f8eb2`](https://github.com/apache/spark/commit/b9f8eb25c99e8270c1108cf76820a45fc0815efb).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740537325
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132424/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740374228
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37002/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741581883
Waiting for SPARK-33705 to fix 3 failed UTs in thriftsever module
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741594159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132456/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744322404
Thank all. Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r537996738
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
This is a good question, but it seems that the filter pushed down in DataSource V1 does not contain the filter related to partition columns too.
The `dataFilters` use to construct `FileSourceScanExec` and pass to `ParquetFileFormat.buildReaderWithPartitionValues` to build `pushed` filters also filter out partition filters, am I right?
https://github.com/apache/spark/blob/e4d1c10760800563d2a30410b46e5b0cd2671c4d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L184-L193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740267417
Thanks for fixing this so quickly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740123891
**[Test build #132381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132381/testReport)** for PR 30652 at commit [`be6cfc8`](https://github.com/apache/spark/commit/be6cfc8db6e3724da865bd80e06b71bdaa81abc2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538011539
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe another way is refer to parquet `FileMetaData#getSchema` to determine which filter should be pushed down
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538173939
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe this is a correct fix. I have a pr to fix partition column is in file.
https://github.com/apache/spark/pull/30670
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740354195
**[Test build #132402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132402/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740505625
**[Test build #132424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132424/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740374231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37002/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740505625
**[Test build #132424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132424/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740532588
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37024/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740546814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37024/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744253304
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37349/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740395302
**[Test build #132402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132402/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538173939
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe this is a correct fix. I have a pr to fix partition column is in file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740395763
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132402/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740537325
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132424/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741510174
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37057/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744191706
**[Test build #132749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132749/testReport)** for PR 30652 at commit [`13bfda3`](https://github.com/apache/spark/commit/13bfda388b2cb4754fb96324fe0fe60a76bcb6b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740395763
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132402/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740033272
**[Test build #132381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132381/testReport)** for PR 30652 at commit [`be6cfc8`](https://github.com/apache/spark/commit/be6cfc8db6e3724da865bd80e06b71bdaa81abc2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740100048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36981/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744253304
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37349/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740546814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37024/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740064436
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36981/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538011539
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe another way is refer to parquet `FileMetaData#getSchema` to determine which filter should be pushed down, but this needs to be done once for each file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538011539
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe another way is to refer to parquet `FileMetaData#getSchema` to determine which filter should be pushed down, but this needs to be done once for each file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum edited a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum edited a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744322404
Thanks all. Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538950386
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Maybe we should use `readDataSchema()` instead of `dataSchema`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740100048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36981/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741594159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132456/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740364869
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37002/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740546792
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37024/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740132835
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132381/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740486957
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741489584
**[Test build #132456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132456/testReport)** for PR 30652 at commit [`b9f8eb2`](https://github.com/apache/spark/commit/b9f8eb25c99e8270c1108cf76820a45fc0815efb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740132835
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132381/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741489584
**[Test build #132456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132456/testReport)** for PR 30652 at commit [`b9f8eb2`](https://github.com/apache/spark/commit/b9f8eb25c99e8270c1108cf76820a45fc0815efb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r538961489
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
Address b9f8eb2 use `readDataSchema()` instead of `dataSchema`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744191706
**[Test build #132749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132749/testReport)** for PR 30652 at commit [`13bfda3`](https://github.com/apache/spark/commit/13bfda388b2cb4754fb96324fe0fe60a76bcb6b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744339933
**[Test build #132749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132749/testReport)** for PR 30652 at commit [`13bfda3`](https://github.com/apache/spark/commit/13bfda388b2cb4754fb96324fe0fe60a76bcb6b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait CheckAnalysis extends PredicateHelper with LookupCatalog `
* `case class UnresolvedView(`
* `case class Decode(params: Seq[Expression], child: Expression) extends RuntimeReplaceable `
* `case class StringDecode(bin: Expression, charset: Expression)`
* `case class NoopCommand(`
* `case class ShowTableExtended(`
* `case class DropView(`
* `case class RepairTable(child: LogicalPlan) extends Command `
* `case class AlterViewSetProperties(`
* `case class AlterViewUnsetProperties(`
* `case class CacheTable(`
* `case class CacheTableAsSelect(`
* `trait BaseCacheTableExec extends V2CommandExec `
* `case class CacheTableExec(`
* `case class CacheTableAsSelectExec(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741510174
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37057/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-744362041
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132749/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #30652:
URL: https://github.com/apache/spark/pull/30652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740354195
**[Test build #132402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132402/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-741581883
Waiting for SPARK-33705 to fix UTs in thriftsever module
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #30652:
URL: https://github.com/apache/spark/pull/30652#discussion_r537956524
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
##########
@@ -50,7 +50,7 @@ case class ParquetScanBuilder(
val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
val isCaseSensitive = sqlConf.caseSensitiveAnalysis
val parquetSchema =
- new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema)
Review comment:
What if the partition column is in file? I guess this might be why we push down partition filter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30652: [SPARK-33673][SQL] Avoid push down partition filters to ParquetScan for DataSourceV2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30652:
URL: https://github.com/apache/spark/pull/30652#issuecomment-740528696
**[Test build #132424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132424/testReport)** for PR 30652 at commit [`fb01c3f`](https://github.com/apache/spark/commit/fb01c3f6d7241813975f2001329bc209ea371083).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org