You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/13 05:14:42 UTC
[GitHub] [spark] huaxingao opened a new pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
huaxingao opened a new pull request #33973:
URL: https://github.com/apache/spark/pull/33973
Co-Authored-By: DB Tsai d_tsai@apple.com
Co-Authored-By: Huaxin Gao huaxin_gao@apple.com
### What changes were proposed in this pull request?
This is the 2nd PR for V2 Filter support. In this PR, we have updated the V2 file source to use V2 Filters. ParquetFilter hasn't been updated to use the V2 Filters yet and will be changed in the next PR.
### Why are the changes needed?
To eliminate the unnecessary conversion between Catalyst types and Scala types when using Filters.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
New and existing test suites
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918326934
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917886672
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47693/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917860262
**[Test build #143191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143191/testReport)** for PR 33973 at commit [`7c4a43b`](https://github.com/apache/spark/commit/7c4a43b27a3fe0717dfc40e90dbdd1f8ddd7a1eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917883586
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47693/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918517248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143210/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917879610
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47693/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918334889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918334889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-919433568
@huaxingao it'd be nice if you can summarize all the changes introduced in this PR, in "What changes were proposed in this pull request" section, which will help the reviewing process. I'm also thinking whether it makes sense to break this down into smaller PRs, for instance, having a separate for translating from V2 to V1 filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917938169
**[Test build #143191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143191/testReport)** for PR 33973 at commit [`7c4a43b`](https://github.com/apache/spark/commit/7c4a43b27a3fe0717dfc40e90dbdd1f8ddd7a1eb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918517248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143210/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918334837
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917938466
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918346449
cc @cloud-fan @sunchao @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917886672
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47693/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918269921
**[Test build #143210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143210/testReport)** for PR 33973 at commit [`fc060c8`](https://github.com/apache/spark/commit/fc060c87224f3e8b6e4a7fb0a19a24b200d0a645).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918269921
**[Test build #143210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143210/testReport)** for PR 33973 at commit [`fc060c8`](https://github.com/apache/spark/commit/fc060c87224f3e8b6e4a7fb0a19a24b200d0a645).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-919584048
I will split this PR into smaller PRs so it will be easier to review. Will close this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-918511698
**[Test build #143210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143210/testReport)** for PR 33973 at commit [`fc060c8`](https://github.com/apache/spark/commit/fc060c87224f3e8b6e4a7fb0a19a24b200d0a645).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917938466
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33973:
URL: https://github.com/apache/spark/pull/33973#issuecomment-917860262
**[Test build #143191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143191/testReport)** for PR 33973 at commit [`7c4a43b`](https://github.com/apache/spark/commit/7c4a43b27a3fe0717dfc40e90dbdd1f8ddd7a1eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao closed pull request #33973: [SPARK-36730][SQL] Use V2 Filter in V2 file source
Posted by GitBox <gi...@apache.org>.
huaxingao closed pull request #33973:
URL: https://github.com/apache/spark/pull/33973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org