You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ueshin <gi...@git.apache.org> on 2014/05/19 10:18:03 UTC
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/825
[SPARK-1880] [SQL] Eliminate unnecessary job executions.
There are unnecessary job executions in `BroadcastNestedLoopJoin`.
When `Innner` or `LeftOuter` join, preparation of `rightOuterMatches` for `RightOuter` or `FullOuter` join is not neccessary.
And when `RightOuter` or `FullOuter`, it should use not `count` and then `reduce` but `fold`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-1880
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/825.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #825
----
commit 7bff4d978fa0cb64ef6d4e935aaa8c394a6fa4ae
Author: Takuya UESHIN <ue...@happy-camper.st>
Date: 2014-05-19T07:47:39Z
Move preparing of rightOuterMatches to after checking joinType not to execute unnecessary job when joinType is Inner or LeftOuter.
commit 63cacbfd41090185de64c27586cacc5265c3c0d7
Author: Takuya UESHIN <ue...@happy-camper.st>
Date: 2014-05-19T07:52:33Z
Use fold instead of count and then reduce to eliminate unnecessary job executions.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43824842
Done & merged #836.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43476605
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43482897
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15079/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43959321
Ok, close this. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43482896
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43842591
@rxin Thanks a lot!
Well, should I close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43741634
@rxin Ah, you mean that we should add right/full outer join support in addition to #734?
I agree with the unfavorable performance of the nested loop join, so we should wait for being merged and then add the right/full outer join support at another issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43727197
We can easily add right outer join support to the hash join though. In general, the nested loop join performs very unfavorably compared with a hash join implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43476612
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43585497
Would be great if you can review that code, @ueshin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43951637
Probably makes sense to close this one now. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43585484
Thanks for submitting this. I actually think this thing is going away to be replaced by broadcast hash join. https://github.com/apache/spark/pull/734
Maybe we should wait for that to be merged? Otherwise a lot of unnecessary conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43599450
@rxin Thank you for your comment.
I checked the code #734, not deeply yet, though.
It seems like broadcast hash join is used only for `Inner` join so broadcast nested loop join will not be replaced.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the pull request:
https://github.com/apache/spark/pull/825#issuecomment-43742049
@rxin BTW, speaking of performance, could you please review the code #836?
I think this is a kind of blocker issue of join strategy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1880] [SQL] Eliminate unnecessary job e...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin closed the pull request at:
https://github.com/apache/spark/pull/825
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---