You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vikram Dixit Kumaraswamy <vi...@gmail.com> on 2013/08/05 19:57:13 UTC
Re: Review Request 12827: HIVE-4611 - SMB joins fail based on bigtable
selection policy.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12827/
-----------------------------------------------------------
(Updated Aug. 5, 2013, 5:57 p.m.)
Review request for hive, Ashutosh Chauhan, Brock Noland, and Gunther Hagleitner.
Changes
-------
Addressed Gunther's comments.
Bugs: HIVE-4611
https://issues.apache.org/jira/browse/HIVE-4611
Repository: hive-git
Description
-------
SMB joins fail based on bigtable selection policy. The default setting for hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big table as the one with largest average partition size. However, this can result in a query failing because this policy conflicts with the big table candidates chosen for outer joins. This policy should just be a tie breaker and not have the ultimate say in the choice of tables.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 12e9334
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java cc9de54
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java 5320143
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BigTableSelectorForAutoSMJ.java db5ff0f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LeftmostBigTableSelectorForAutoSMJ.java db3c9e7
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java cd1b4ad
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java b882f87
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 3071713
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java e214807
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java da5115b
ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q c858254
ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q PRE-CREATION
ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out 0eddb69
ql/src/test/results/clientnegative/smb_bucketmapjoin.q.out 7a5b8c1
ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/12827/diff/
Testing
-------
All tests pass on hadoop 1.
Thanks,
Vikram Dixit Kumaraswamy
Re: Review Request 12827: HIVE-4611 - SMB joins fail based on bigtable
selection policy.
Posted by Vikram Dixit Kumaraswamy <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12827/
-----------------------------------------------------------
(Updated Aug. 14, 2013, 7:21 p.m.)
Review request for hive, Ashutosh Chauhan, Brock Noland, and Gunther Hagleitner.
Changes
-------
Addressed Ashutosh's comments.
Bugs: HIVE-4611
https://issues.apache.org/jira/browse/HIVE-4611
Repository: hive-git
Description
-------
SMB joins fail based on bigtable selection policy. The default setting for hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big table as the one with largest average partition size. However, this can result in a query failing because this policy conflicts with the big table candidates chosen for outer joins. This policy should just be a tie breaker and not have the ultimate say in the choice of tables.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 12e9334
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java fda2f84
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java 1bed28f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BigTableSelectorForAutoSMJ.java db5ff0f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LeftmostBigTableSelectorForAutoSMJ.java db3c9e7
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java cd1b4ad
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java d33ea91
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 3071713
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java e214807
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java da5115b
ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q c858254
ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q PRE-CREATION
ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out 0eddb69
ql/src/test/results/clientnegative/smb_bucketmapjoin.q.out 7a5b8c1
ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/12827/diff/
Testing
-------
All tests pass on hadoop 1.
Thanks,
Vikram Dixit Kumaraswamy