You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/24 10:34:13 UTC
[GitHub] [spark] beliefer opened a new pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
beliefer opened a new pull request #33822:
URL: https://github.com/apache/spark/pull/33822
### What changes were proposed in this pull request?
Spark SQL includes a data source that can read data from other databases using JDBC.
Spark also supports the case-insensitive option `pushDownPredicate`.
According to http://spark.apache.org/docs/latest/sql-data-sources-jdbc.html, If set `pushDownPredicate` to false, no filter will be pushed down to the JDBC data source and thus all filters will be handled by Spark.
But I find it still be pushed down to JDBC data source.
### Why are the changes needed?
Fix bug `pushDownPredicate`=false failed to prevent push down filters to JDBC data source.
### Does this PR introduce _any_ user-facing change?
'No'.
The output of query will not change.
### How was this patch tested?
Jenkins test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906301449
**[Test build #142812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142812/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906107754
@beliefer I see.
But at this line `assert(relation.unhandledFilters(FiltersPushed.list.toArray).toSet === expectedUnhandledFilters)` https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala#L331, why are we getting `unhandledFilters` from `FiltersPushed.list`? Should we get `unhandledFilters` from the original filters?
I still think we shouldn't push down `unhandledFilters` to the data source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904582746
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905595201
I took a look at v2 path. Seems to me that filter push down logic in v2 is different from v1.
V2
`val (pushed, unSupported) = filters.partition(JDBCRDD.compileFilter(_, dialect).isDefined)`
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala#L51
We only push down the supported filters, the unSupported is returned as the post scan filters.
V1:
`val (unhandledPredicates, pushedFilters, handledFilters) = selectFilters(relation.relation, candidatePredicates)`
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L388
We push down the `pushedFilters`, not the `handledFilters`. https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L436
Seems to me that we should push down handledFilters, not the translated filters (`pushedFilters`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906219484
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47296/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906172148
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906267974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906904324
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905272743
**[Test build #142748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142748/testReport)** for PR 33822 at commit [`d34cdfd`](https://github.com/apache/spark/commit/d34cdfd3670a3c3be1f9128fa79a3eb5a55fd73d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906983426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142830/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906143364
**[Test build #142790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142790/testReport)** for PR 33822 at commit [`91c2284`](https://github.com/apache/spark/commit/91c22843da85a7a40f0d558e880563374d95be51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904901221
**[Test build #142725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142725/testReport)** for PR 33822 at commit [`0fecee8`](https://github.com/apache/spark/commit/0fecee8732f21e219fd00a948f92af5c8f4c7771).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906239573
**[Test build #142806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142806/testReport)** for PR 33822 at commit [`24d3c94`](https://github.com/apache/spark/commit/24d3c94cd88c41b41e446efd2332b0005155cac2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906367577
**[Test build #142796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142796/testReport)** for PR 33822 at commit [`ecb3a47`](https://github.com/apache/spark/commit/ecb3a47b224c0d4e7cc40c6f26f9a7f3d5b9aee3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906379826
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142796/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906211323
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906903649
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906219085
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906125147
>why are we getting `unhandledFilters` from `FiltersPushed.list`?
@huaxingao I don't know the reason. If we shouldn't push down unhandledFilters to the data source, we should adust these test cases too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906255159
**[Test build #142790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142790/testReport)** for PR 33822 at commit [`91c2284`](https://github.com/apache/spark/commit/91c22843da85a7a40f0d558e880563374d95be51).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906532321
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142812/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906267974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906982223
**[Test build #142830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142830/testReport)** for PR 33822 at commit [`4b4f78a`](https://github.com/apache/spark/commit/4b4f78ababd83664340722ea47b24352dda284ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906259717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905161479
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47248/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906904324
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906277038
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906454108
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906031199
> Seems to me that we should push down handledFilters, not the translated filters (`pushedFilters`)
@huaxingao I tried on the way, but some test case cannot pass.
Please see https://github.com/apache/spark/blob/159ff9fd14f7e0581833428c495c0e2c34f7e320/sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala#L81
According to the test case, it seems that the pushed filters should as the parameters of `def buildScan`.
cc @cloud-fan @gengliangwang
In fact, I commit the update https://github.com/apache/spark/pull/33822/commits/0fecee8732f21e219fd00a948f92af5c8f4c7771 in previous like your consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906983426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142830/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906031199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r696414946
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -409,7 +409,7 @@ object DataSourceStrategy
pushedFilters.toSet,
handledFilters,
None,
- scanBuilder(requestedColumns, candidatePredicates, pushedFilters),
+ scanBuilder(requestedColumns, candidatePredicates, handledFilters.toSeq),
Review comment:
I think this change is correct but is pretty risky. DS v1 has been there for many years and it's possible that some v1 sources forget to override `unhandledFilters`.
Before this PR, it's not a big deal if v1 sources forget to override `unhandledFilters`. Filter pushdown still works, only the EXPLAIN result will be inaccurate.
I don't think we can make this change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-908839974
@cloud-fan Thanks a lot! @gengliangwang @huaxingao Thank you for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r695845556
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
##########
@@ -278,12 +278,18 @@ private[sql] case class JDBCRelation(
}
override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ // When pushDownPredicate is false, all Filters that need to be pushed down should be ignored
+ val pushedFilters = if (jdbcOptions.pushDownPredicate) {
+ filters
+ } else {
+ Array.empty[Filter]
+ }
Review comment:
If `pushDownPredicate` is false, the unhandledFilters are set to all the filters here https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala#L276. Seems to me that the unhandledFilters shouldn't be pushed down to JDBC at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906277069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906143364
**[Test build #142790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142790/testReport)** for PR 33822 at commit [`91c2284`](https://github.com/apache/spark/commit/91c22843da85a7a40f0d558e880563374d95be51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33822:
URL: https://github.com/apache/spark/pull/33822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906294384
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47310/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r695853027
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
##########
@@ -278,12 +278,18 @@ private[sql] case class JDBCRelation(
}
override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
+ // When pushDownPredicate is false, all Filters that need to be pushed down should be ignored
+ val pushedFilters = if (jdbcOptions.pushDownPredicate) {
+ filters
+ } else {
+ Array.empty[Filter]
+ }
Review comment:
Yea, it's a bug since day 1: https://github.com/apache/spark/pull/21875
I think we should update the tested add at that time and check the real pushed filters in JDBC source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904582746
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906532321
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142812/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906507481
**[Test build #142812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142812/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-908839974
@cloud-fan Thanks a lot! @gengliangwang @huaxingao Thank you for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906267930
**[Test build #142809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142809/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906250847
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47296/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906031199
> Seems to me that we should push down handledFilters, not the translated filters (`pushedFilters`)
@huaxingao I tried on the way, but some test case cannot pass.
Please see https://github.com/apache/spark/blob/159ff9fd14f7e0581833428c495c0e2c34f7e320/sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala#L81
According the test case, it seems that the pushed filters should as the parameters of `def buildScan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906452034
**[Test build #142806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142806/testReport)** for PR 33822 at commit [`24d3c94`](https://github.com/apache/spark/commit/24d3c94cd88c41b41e446efd2332b0005155cac2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906379826
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142796/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906297268
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904901803
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142725/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906885161
**[Test build #142830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142830/testReport)** for PR 33822 at commit [`4b4f78a`](https://github.com/apache/spark/commit/4b4f78ababd83664340722ea47b24352dda284ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r696583283
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
##########
@@ -1723,6 +1723,34 @@ class JDBCSuite extends QueryTest
Row("fred", 1) :: Nil)
}
+ test(
+ "SPARK-36574: pushDownPredicate=false should prevent push down filters to JDBC data source") {
+ val df = spark.read.format("jdbc")
+ .option("Url", urlWithUserAndPass)
+ .option("dbTable", "test.people")
+ val df1 = df
+ .option("pushDownPredicate", false)
+ .load()
+ .filter("theid = 1")
+ .select("name", "theid")
+ val df2 = df
+ .load()
+ .select("name", "theid")
+
+ def getRowCount(df: DataFrame): Long = {
+ val queryExecution = df.queryExecution
+ val rawPlan = queryExecution.executedPlan.collect {
+ case p: DataSourceScanExec => p
+ } match {
+ case Seq(p) => p
+ case _ => fail(s"More than one PhysicalRDD found\n$queryExecution")
+ }
+ rawPlan.execute().count()
+ }
+
+ assert(getRowCount(df1) == getRowCount(df2))
Review comment:
let's also test if we enable `pushDownPredicate`, the row count is smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905144267
**[Test build #142748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142748/testReport)** for PR 33822 at commit [`d34cdfd`](https://github.com/apache/spark/commit/d34cdfd3670a3c3be1f9128fa79a3eb5a55fd73d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906263306
**[Test build #142809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142809/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r696582937
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
##########
@@ -1723,6 +1723,34 @@ class JDBCSuite extends QueryTest
Row("fred", 1) :: Nil)
}
+ test(
+ "SPARK-36574: pushDownPredicate=false should prevent push down filters to JDBC data source") {
+ val df = spark.read.format("jdbc")
+ .option("Url", urlWithUserAndPass)
+ .option("dbTable", "test.people")
+ val df1 = df
+ .option("pushDownPredicate", false)
+ .load()
+ .filter("theid = 1")
+ .select("name", "theid")
+ val df2 = df
+ .load()
+ .select("name", "theid")
+
+ def getRowCount(df: DataFrame): Long = {
+ val queryExecution = df.queryExecution
+ val rawPlan = queryExecution.executedPlan.collect {
+ case p: DataSourceScanExec => p
+ } match {
+ case Seq(p) => p
+ case _ => fail(s"More than one PhysicalRDD found\n$queryExecution")
+ }
+ rawPlan.execute().count()
+ }
+
+ assert(getRowCount(df1) == getRowCount(df2))
Review comment:
nit: `assert(getRowCount(df1) == df2.count)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906298915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47310/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906298915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47310/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-908175872
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905332834
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906180712
**[Test build #142796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142796/testReport)** for PR 33822 at commit [`ecb3a47`](https://github.com/apache/spark/commit/ecb3a47b224c0d4e7cc40c6f26f9a7f3d5b9aee3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905332834
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-908253112
thanks, merging to master/3.2!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906277069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906219085
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905382766
> is it a long-standing bug? or a regression which was introduced recently? and does JDBC v2 have the same problem?
I doubt it's a long-standing bug. I checked JDBC v2 have't the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906031199
> Seems to me that we should push down handledFilters, not the translated filters (`pushedFilters`)
@huaxingao I tried on the way, but some test case cannot pass.
Please see https://github.com/apache/spark/blob/159ff9fd14f7e0581833428c495c0e2c34f7e320/sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala#L81
According to the test case, it seems that the pushed filters should as the parameters of `def buildScan`.
cc @cloud-fan @gengliangwang
In fact, I commit the update https://github.com/apache/spark/pull/33822/commits/0fecee8732f21e219fd00a948f92af5c8f4c7771 like your consideration in previous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906334616
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905355211
ping @cloud-fan @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905144267
**[Test build #142748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142748/testReport)** for PR 33822 at commit [`d34cdfd`](https://github.com/apache/spark/commit/d34cdfd3670a3c3be1f9128fa79a3eb5a55fd73d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905274071
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142748/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906263306
**[Test build #142809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142809/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-908175872
ping @cloud-fan cc @huaxingao @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906275659
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906900848
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904582712
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905486750
@beliefer can we add new test case for the fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906885161
**[Test build #142830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142830/testReport)** for PR 33822 at commit [`4b4f78a`](https://github.com/apache/spark/commit/4b4f78ababd83664340722ea47b24352dda284ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906301449
**[Test build #142812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142812/testReport)** for PR 33822 at commit [`9c01364`](https://github.com/apache/spark/commit/9c01364e1850cd6274226deb93abaa898bcbd01e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904531105
**[Test build #142725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142725/testReport)** for PR 33822 at commit [`0fecee8`](https://github.com/apache/spark/commit/0fecee8732f21e219fd00a948f92af5c8f4c7771).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906454108
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142806/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905175319
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47248/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r696424411
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -409,7 +409,7 @@ object DataSourceStrategy
pushedFilters.toSet,
handledFilters,
None,
- scanBuilder(requestedColumns, candidatePredicates, pushedFilters),
+ scanBuilder(requestedColumns, candidatePredicates, handledFilters.toSeq),
Review comment:
OK. Let me revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906239573
**[Test build #142806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142806/testReport)** for PR 33822 at commit [`24d3c94`](https://github.com/apache/spark/commit/24d3c94cd88c41b41e446efd2332b0005155cac2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904531105
**[Test build #142725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142725/testReport)** for PR 33822 at commit [`0fecee8`](https://github.com/apache/spark/commit/0fecee8732f21e219fd00a948f92af5c8f4c7771).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905179604
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47248/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905366054
is it a long-standing bug? or a regression which was introduced recently? and does JDBC v2 have the same problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904901803
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142725/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906180712
**[Test build #142796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142796/testReport)** for PR 33822 at commit [`ecb3a47`](https://github.com/apache/spark/commit/ecb3a47b224c0d4e7cc40c6f26f9a7f3d5b9aee3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33822:
URL: https://github.com/apache/spark/pull/33822#discussion_r696420571
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
##########
@@ -315,6 +315,19 @@ class JDBCSuite extends QueryTest
assert(parentPlan.isInstanceOf[org.apache.spark.sql.execution.WholeStageCodegenExec])
val node = parentPlan.asInstanceOf[org.apache.spark.sql.execution.WholeStageCodegenExec]
assert(node.child.isInstanceOf[org.apache.spark.sql.execution.FilterExec])
+ val relation = df.queryExecution.analyzed.collectFirst {
+ case LogicalRelation(r, _, _, _) => r
+ }.get
+ assert(relation.isInstanceOf[org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation])
+ val jdbcRelation =
+ relation.asInstanceOf[org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation]
+ if (jdbcRelation.jdbcOptions.pushDownPredicate == false) {
Review comment:
The method name is `checkNotPushdown`, do we need this `if`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905179604
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47248/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906339227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906259717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-905274071
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142748/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false should prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906339227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-904559603
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47225/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on pull request #33822: [SPARK-36574][SQL] pushDownPredicate=false failed to prevent push down filters to JDBC data source
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on pull request #33822:
URL: https://github.com/apache/spark/pull/33822#issuecomment-906031199
> Seems to me that we should push down handledFilters, not the translated filters (`pushedFilters`)
@huaxingao I tried on the way, but some test case cannot pass.
Please see https://github.com/apache/spark/blob/159ff9fd14f7e0581833428c495c0e2c34f7e320/sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala#L81
According to the test case, it seems that the pushed filters should as the parameters of `def buildScan`.
cc @cloud-fan @gengliangwang
In fact, I commit the update https://github.com/apache/spark/pull/33822/commits/0fecee8732f21e219fd00a948f92af5c8f4c7771 like your consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org