You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by jihoonson <gi...@git.apache.org> on 2015/08/20 10:49:50 UTC
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
GitHub user jihoonson opened a pull request:
https://github.com/apache/tajo/pull/706
TAJO-1766: Improve the performance of cross join
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jihoonson/tajo-2 TAJO-1766
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/706.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #706
----
commit c585fd9b9f126cb3cd0492a8c8cfe07d82f617db
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-16T14:28:15Z
Need to consider cross and non-cross joins together
commit 18fc1356c0d5296c49b614402f90fc84c4ac0790
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-17T01:36:02Z
Merge branch 'TAJO-1766' of https://github.com/jihoonson/tajo-2 into TAJO-1766
commit b2c8435a886c8008598c367e5e757b601148cfac
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-17T06:18:01Z
TAJO-1766
commit 1f974ceb14a62310ade9a7f631ee3ad531f080de
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-17T09:10:12Z
TAJO-1766
commit 90306b6495314c28f4f7bdc1cb54b1ea799c689d
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-17T10:21:15Z
TAJO-1766
commit 989d0ebd20e4250cd236029a796614f1e64ad891
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-18T01:51:36Z
TAJO-1766
commit 25a7bccf9be42ebea99e8fae4c202b5332d9310e
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-19T01:05:42Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1766
commit 96dd4329d1639615f4ffa65b9cf2562d0853e4e0
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-19T10:13:47Z
add post logical plan verification
commit 5014b05277d9b79515fcff2d7253ad2eec2d650f
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-20T01:35:47Z
improve error message
commit ccc6eba2f445f2b813322b4fb806b829bd0f3753
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-20T08:47:19Z
Remove estimation code
commit 55dc2c6f3cfdd86ae10a1babcacc01b5e8bd6d03
Author: Jihoon Son <ji...@apache.org>
Date: 2015-08-20T08:48:33Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1766
Conflicts:
tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-134537896
Thanks guys!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-134164003
This patch is ready for review. Sorry for a large patch.
Here are some highlights of changes.
* Added a session variable to set the limitation of broadcast table size for cross join. This value is valid only when ```TEST_BROADCAST_JOIN_ENABLED``` is set.
* Cross join is always executed with broadcast join. To do so, at least one input of cross join should be the relation which is smaller than ```BROADCAST_CROSS_JOIN_THRESHOLD```.
* Added ```PostLogicalPlanVerifier``` to verify that cross join is executable or not.
* Fixed some bugs in BroadcastJoinRule.
* Fixed some bugs in QueryTestCaseBase.
* Removed BNL and NL join executors. Instead, each task executes cross join with hash join. This is because one of inputs of cross join is always cached in the broadcast cache holder.
* Improved unique key generation for a scan executor in ```TaskAttemptContext```.
I've tested cross join performance with a cluster consisting of a master and 5 workers. Each worker equips 48 cores, 80 GB memory, and 24 disks.
#### Data
* partsupp: 80000000 rows (12.2 GB in TEXT file)
* supplier_small: 100000 rows (14.1 MB in TEXT file)
#### Query
```
select count(*) from partsupp, supplier_small
```
#### Result
1 hrs, 14 mins, 33 sec is taken with this patch. The above query runs forever without this patch because a single worker executes cross join.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-134536104
Here is my +1. The patch looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by jinossy <gi...@git.apache.org>.
Github user jinossy commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-134512935
Awesome!!
I've test your query on Tez and same cluster. but I've stop the query after 5h
```
hive> select count(*) from partsupp, supplier_small;
Warning: Shuffle Join MERGEJOIN[11][tables = [partsupp, supplier_small]] in Stage 'Reducer 2' is a cross product
Query ID = hadoop_20150825100744_8e11a2d9-d918-473f-bad2-36e403d6eca3
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1440464539141_0001)
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Exiting the JVM. SUCCEEDED 224 224 0 0 0 0
Interrupting... Be patient, this might take some time.0 0 0 0
Map 1 .......... SUCCEEDED 224 224 0 0 0 0
Map 4 .......... SUCCEEDED 1 1 0 0 0 0
Reducer 2 RUNNING 1 0 1 0 0 0
Reducer 3 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES: 02/04 [=========================>>-] 99% ELAPSED TIME: 22510.28 s
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-133716586
This patch is not ready for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/706#issuecomment-132942635
Please don't start review until #702 is committed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1766: Improve the performance of cross joi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/706
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---