You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2015/12/27 20:03:18 UTC
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/10490
[SPARK-12532] [SQL] Join-key Pushdown via Predicate Transitivity
More predicates in join conditions/filters can be pushed down via predicate transitivity. More predicate pushdown could greatly improve the join performance.
For example, we can infer the extra predicate `upperCaseData.N = 3` in the following query:
```
"SELECT * FROM upperCaseData JOIN lowerCaseData where lowerCaseData.n = upperCaseData.N and lowerCaseData.n = 3"
```
Before the improvement, the optimized logical plan is
```
== Optimized Logical Plan ==
Project [N#16,L#17,n#18,l#19]
+- Join Inner, Some((n#18 = N#16))
:- LogicalRDD [N#16,L#17], MapPartitionsRDD[17] at beforeAll at BeforeAndAfterAll.scala:187
+- Filter (n#18 = 3)
+- LogicalRDD [n#18,l#19], MapPartitionsRDD[19] at beforeAll at BeforeAndAfterAll.scala:187
```
After the improvement, the optimized logical plan should be like
```
== Optimized Logical Plan ==
Project [N#16,L#17,n#18,l#19]
+- Join Inner, Some((n#18 = N#16))
:- Filter (N#16 = 3)
: +- LogicalRDD [N#16,L#17], MapPartitionsRDD[17] at beforeAll at BeforeAndAfterAll.scala:187
+- Filter (n#18 = 3)
+- LogicalRDD [n#18,l#19], MapPartitionsRDD[19] at beforeAll at BeforeAndAfterAll.scala:187
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark conditionInfer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10490
----
commit fb84dbac240361e086fdb0d4a18760006c0d4e76
Author: gatorsmile <ga...@gmail.com>
Date: 2015-12-27T18:42:04Z
Infer and push down join/filter conditions
commit 918ea2ce80d2b0f749349ff58fe4967760388814
Author: gatorsmile <ga...@gmail.com>
Date: 2015-12-27T18:42:29Z
Merge remote-tracking branch 'upstream/master' into conditionInfer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/10490#issuecomment-170758119
Thanks, close it now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10490#issuecomment-167440556
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48355/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10490#issuecomment-167436825
**[Test build #48355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48355/consoleFull)** for PR 10490 at commit [`918ea2c`](https://github.com/apache/spark/commit/918ea2ce80d2b0f749349ff58fe4967760388814).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/10490
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10490#issuecomment-167440555
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10490#issuecomment-167440524
**[Test build #48355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48355/consoleFull)** for PR 10490 at commit [`918ea2c`](https://github.com/apache/spark/commit/918ea2ce80d2b0f749349ff58fe4967760388814).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org