You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2016/03/16 18:36:33 UTC
[GitHub] spark pull request: Constraint filter
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11765
Constraint filter
#### What changes were proposed in this pull request?
In the query, Optimizer generates many duplicate constraints in Filter constraints by applying the rule PushPredicateThroughProject.
```SQL
SELECT unionsrc1.key, unionsrc1.value, unionsrc2.key, unionsrc2.value FROM (select 'tst1' as key, cast(count(1) as string) as value from parquet_t1 s1 UNION ALL select s2.key as key, s2.value as value from parquet_t1 s2 where s2.key < 10) unionsrc1 JOIN (select 'tst1' as key, cast(count(1) as string) as value from parquet_t1 s3 UNION ALL select s4.key as key, s4.value as value from parquet_t1 s4 where s4.key < 10) unionsrc2 ON (unionsrc1.key = unionsrc2.key)
```
Due to this issue, it also hits the max iteration, as shown in the log output https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53176/consoleFull.
This PR uses the constraints to avoid pushing any predicate that already exists in its child's Constraints. Also, it will not pushing any predicate that does not contain any reference, since it could introduce the same issue.
Will introduce the same idea in the similar rules:
- PushPredicateThroughJoin,
- PushPredicateThroughGenerate,
- PushPredicateThroughAggregate,
- SetOperationPushDown
Should I do it in the same PR? or different PRs? @marmbrus
#### How was this patch tested?
Added a test case and also manually tested the case that causes the exception in https://github.com/apache/spark/pull/11714
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark constraintFilter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11765.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11765
----
commit 684b36a2889a196056cef2edd73a600726b0e627
Author: gatorsmile <ga...@gmail.com>
Date: 2016-03-16T16:38:29Z
no push down for constant predicates and the predicates that child contains.
commit 6a0fd8aed6e7b0c0b9a855512b3508dc36531029
Author: gatorsmile <ga...@gmail.com>
Date: 2016-03-16T17:25:36Z
no push down for constant predicates and the predicates that child contains.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-199444598
@sameeragarwal That is just an example that exposes this issue. I think `Cast` is just one of multiple compound cases. If we can avoid pushing down any filters that have been included in the children's `Constraints`. Then, it will avoid useless predicate pushdown.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197503845
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11765: [SPARK-13936] [SQL] Avoid Predicate Pushdown Usin...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/11765
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11765: [SPARK-13936] [SQL] Avoid Predicate Pushdown Using Const...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/11765
Same, is this still an issue, active?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-199423757
@gatorsmile isn't the issue that you pointed out in the PR description just an artifact of union not propagating constraints correctly when it encounters a `Cast` in a child?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197451969
**[Test build #53329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53329/consoleFull)** for PR 11765 at commit [`da6238e`](https://github.com/apache/spark/commit/da6238e7aff8f23e9e7d6cc0f869074e39f7562a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Constraint filter
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197448448
**[Test build #53328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53328/consoleFull)** for PR 11765 at commit [`6a0fd8a`](https://github.com/apache/spark/commit/6a0fd8aed6e7b0c0b9a855512b3508dc36531029).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-198591713
@davies, changing `InferFiltersFromConstraints` is unable to fix all the issues. `InferFiltersFromConstraints` is to infer the filter conditions based on the Constraints of a specific node. Fixing `InferFiltersFromConstraints` is needed but it does not cover all the cases. If you checking the Constraint generation, it will not pop up all the conditions to their parents. It depends on the operator type. This is the fundamental reason why we should use the Constraints to avoid filter condition pushdown.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197495711
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197495717
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53328/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-198534808
@gatorsmile Should we fix this in InferFiltersFromConstraints itself?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197495421
**[Test build #53328 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53328/consoleFull)** for PR 11765 at commit [`6a0fd8a`](https://github.com/apache/spark/commit/6a0fd8aed6e7b0c0b9a855512b3508dc36531029).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11765: [SPARK-13936] [SQL] Avoid Predicate Pushdown Using Const...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/11765
The issue has been resolved. Let me close it. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197503850
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53329/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197502966
**[Test build #53329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53329/consoleFull)** for PR 11765 at commit [`da6238e`](https://github.com/apache/spark/commit/da6238e7aff8f23e9e7d6cc0f869074e39f7562a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13936] [SQL] Avoid Predicate Pushdown U...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11765#issuecomment-197514258
cc @marmbrus @sameeragarwal @cloud-fan @nongli
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org