You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Szehon Ho <sz...@cloudera.com> on 2015/01/30 04:27:43 UTC
Review Request 30443: HIVE-9192 : One-pass SMB Optimizations [Spark
Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30443/
-----------------------------------------------------------
Review request for hive and Xuefu Zhang.
Repository: hive-git
Description
-------
This patch refactors SMB MapJoin optimizations in Spark to be one-pass. The main part of SMB MapJoin optimization is to annotate the MapWork with the information from SMBMapJoinOperator and its roots (TableScans).
Instead of doing MapWork init/annotation in the SparkSortMergeJoinFactory in a second pass, now both GenSparkWork and SparkSortMergeJoinFactory classes collect information. After the one-pass, we go through all the SMBJoinOperators and annotate their mapworks.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java 6e0ac38
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 773cfbd
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 0eac6e1
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java cb5d4fe
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 3a7477a
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkSMBMapJoinInfo.java PRE-CREATION
Diff: https://reviews.apache.org/r/30443/diff/
Testing
-------
Thanks,
Szehon Ho
Re: Review Request 30443: HIVE-9192 : One-pass SMB Optimizations
[Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30443/#review70389
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Jan. 30, 2015, 3:30 a.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30443/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2015, 3:30 a.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-9192
> https://issues.apache.org/jira/browse/HIVE-9192
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch refactors SMB MapJoin optimizations in Spark to be one-pass. The main part of SMB MapJoin optimization is to annotate the MapWork with the information from SMBMapJoinOperator and its roots (TableScans).
>
> Instead of doing MapWork init/annotation in the SparkSortMergeJoinFactory in a second pass, now both GenSparkWork and SparkSortMergeJoinFactory classes collect information. After the one-pass, we go through all the SMBJoinOperators and annotate their mapworks.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java 6e0ac38
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 773cfbd
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 0eac6e1
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java cb5d4fe
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 3a7477a
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkSMBMapJoinInfo.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30443/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Szehon Ho
>
>
Re: Review Request 30443: HIVE-9192 : One-pass SMB Optimizations
[Spark Branch]
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30443/
-----------------------------------------------------------
(Updated Jan. 30, 2015, 3:30 a.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Adding JIRA number.
Bugs: HIVE-9192
https://issues.apache.org/jira/browse/HIVE-9192
Repository: hive-git
Description
-------
This patch refactors SMB MapJoin optimizations in Spark to be one-pass. The main part of SMB MapJoin optimization is to annotate the MapWork with the information from SMBMapJoinOperator and its roots (TableScans).
Instead of doing MapWork init/annotation in the SparkSortMergeJoinFactory in a second pass, now both GenSparkWork and SparkSortMergeJoinFactory classes collect information. After the one-pass, we go through all the SMBJoinOperators and annotate their mapworks.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java 6e0ac38
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 773cfbd
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 0eac6e1
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java cb5d4fe
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 3a7477a
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkSMBMapJoinInfo.java PRE-CREATION
Diff: https://reviews.apache.org/r/30443/diff/
Testing
-------
Thanks,
Szehon Ho