You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rui Li <ru...@intel.com> on 2014/10/28 15:02:24 UTC
Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/
-----------------------------------------------------------
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-8610
https://issues.apache.org/jira/browse/HIVE-8610
Repository: hive-git
Description
-------
This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27283/diff/
Testing
-------
Thanks,
Rui Li
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Rui Li <ru...@intel.com>.
> On Oct. 29, 2014, 5:09 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java, line 626
> > <https://reviews.apache.org/r/27283/diff/2/?file=739757#file739757line626>
> >
> > static method? Maybe we can create such clone() method in QBJoinTree class.
Yes I was thinking add such a method to QBJoinTree. Do you think we should make QBJoinTree clonable and implement deep copy?
> On Oct. 29, 2014, 5:09 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java, line 694
> > <https://reviews.apache.org/r/27283/diff/2/?file=739757#file739757line694>
> >
> > The method seems to be recursive, but I'm not sure it's correct. We call the same method for the parents and children of the current operator. When a parent or child operator of the current operator is processed, wouldn't the process go back to the current operator again? Recursion should be done in one direction and should have clear end condition.
Thanks a lot for the catch. The method can work the way it's being used. But you're right, I will make it more robust.
- Rui
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59001
-----------------------------------------------------------
On Oct. 29, 2014, 2:59 a.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2014, 2:59 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59001
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100281>
static method? Maybe we can create such clone() method in QBJoinTree class.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100282>
Static method? Same to below
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100279>
The method seems to be recursive, but I'm not sure it's correct. We call the same method for the parents and children of the current operator. When a parent or child operator of the current operator is processed, wouldn't the process go back to the current operator again? Recursion should be done in one direction and should have clear end condition.
- Xuefu Zhang
On Oct. 29, 2014, 2:59 a.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2014, 2:59 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Rui Li <ru...@intel.com>.
> On Oct. 30, 2014, 2:29 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java, line 694
> > <https://reviews.apache.org/r/27283/diff/2/?file=739757#file739757line694>
> >
> > Maybe we should have a different operator clone. Check.
> >
> > There is a method (though not called by Operator.clone()) in OperatorFactor, which takes care of colMap:
> >
> > public static <T extends OperatorDesc> Operator<T> getAndMakeChild(T conf,
> > RowSchema rwsch, Map<String, ExprNodeDesc> colExprMap, List<Operator<? extends OperatorDesc>> oplist)
Thanks Xuefu. I can try change Operatot.clone to call this method.
- Rui
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59116
-----------------------------------------------------------
On Oct. 29, 2014, 2:59 a.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2014, 2:59 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59116
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100434>
Yeah. Obviously there is a need.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100435>
Maybe we should have a different operator clone. Check.
There is a method (though not called by Operator.clone()) in OperatorFactor, which takes care of colMap:
public static <T extends OperatorDesc> Operator<T> getAndMakeChild(T conf,
RowSchema rwsch, Map<String, ExprNodeDesc> colExprMap, List<Operator<? extends OperatorDesc>> oplist)
- Xuefu Zhang
On Oct. 29, 2014, 2:59 a.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2014, 2:59 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59188
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Oct. 30, 2014, 1:07 p.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 1:07 p.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 273691e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/java/org/apache/hadoop/hive/ql/parse/JoinCond.java 96df830
> ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java 94c563f
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Rui Li <ru...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/
-----------------------------------------------------------
(Updated Oct. 30, 2014, 1:07 p.m.)
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-8610
https://issues.apache.org/jira/browse/HIVE-8610
Repository: hive-git
Description
-------
This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 273691e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
ql/src/java/org/apache/hadoop/hive/ql/parse/JoinCond.java 96df830
ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java 94c563f
ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27283/diff/
Testing
-------
Thanks,
Rui Li
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/#review59186
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java
<https://reviews.apache.org/r/27283/#comment100476>
1. Let's only catch CloneNotSupportedException here, and let caller handle other exceptions.
2. Don't call e.printStacktrace(). you can log it in the log message LOG.debug("message", e) if necessary.
- Xuefu Zhang
On Oct. 30, 2014, 8:47 a.m., Rui Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27283/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 8:47 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-8610
> https://issues.apache.org/jira/browse/HIVE-8610
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
> The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 273691e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
> ql/src/java/org/apache/hadoop/hive/ql/parse/JoinCond.java 96df830
> ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java 94c563f
> ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
> ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27283/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Rui Li
>
>
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Rui Li <ru...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/
-----------------------------------------------------------
(Updated Oct. 30, 2014, 8:47 a.m.)
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-8610
https://issues.apache.org/jira/browse/HIVE-8610
Repository: hive-git
Description
-------
This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 273691e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
ql/src/java/org/apache/hadoop/hive/ql/parse/JoinCond.java 96df830
ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java 94c563f
ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27283/diff/
Testing
-------
Thanks,
Rui Li
Re: Review Request 27283: Compile time skew join optimization doesn't
work with auto map join
Posted by Rui Li <ru...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27283/
-----------------------------------------------------------
(Updated Oct. 29, 2014, 2:59 a.m.)
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-8610
https://issues.apache.org/jira/browse/HIVE-8610
Repository: hive-git
Description
-------
This patch adds QBJoinTree and colExprMap for the cloned join operator tree in SkewJoinOptimizer, so that CommonJoinResolver can properly convert the cloned join to map join.
The added tests are copied from skewjoinopt*.q, except that auto map join is enabled.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java e87c41b
ql/src/test/queries/clientpositive/skewjoin_mapjoin1.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin10.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin11.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin2.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin3.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin4.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin5.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin6.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin7.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin8.q PRE-CREATION
ql/src/test/queries/clientpositive/skewjoin_mapjoin9.q PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin1.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin10.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin2.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin3.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin4.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin8.q.out PRE-CREATION
ql/src/test/results/clientpositive/skewjoin_mapjoin9.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/27283/diff/
Testing
-------
Thanks,
Rui Li