You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chao Sun <ch...@cloudera.com> on 2014/11/14 01:03:49 UTC
Review Request 28017: HIVE-8776 - Generate MapredLocalWork in
SparkMapJoinResolver [Spark Brach]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28017/
-----------------------------------------------------------
Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
Bugs: HIVE-8776
https://issues.apache.org/jira/browse/HIVE-8776
Repository: hive-git
Description
-------
In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables.
We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 4b9a6cb
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0
Diff: https://reviews.apache.org/r/28017/diff/
Testing
-------
Thanks,
Chao Sun
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in
SparkMapJoinResolver [Spark Brach]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28017/#review61393
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Nov. 14, 2014, 2:43 a.m., Chao Sun wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28017/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2014, 2:43 a.m.)
>
>
> Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
>
>
> Bugs: HIVE-8776
> https://issues.apache.org/jira/browse/HIVE-8776
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables.
> We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 9ce1a18
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0
>
> Diff: https://reviews.apache.org/r/28017/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chao Sun
>
>
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in
SparkMapJoinResolver [Spark Brach]
Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28017/
-----------------------------------------------------------
(Updated Nov. 14, 2014, 2:43 a.m.)
Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
Changes
-------
Thanks Xuefu for the comments!
Bugs: HIVE-8776
https://issues.apache.org/jira/browse/HIVE-8776
Repository: hive-git
Description
-------
In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables.
We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 9ce1a18
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0
Diff: https://reviews.apache.org/r/28017/diff/
Testing
-------
Thanks,
Chao Sun
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in
SparkMapJoinResolver [Spark Brach]
Posted by Chao Sun <ch...@cloudera.com>.
> On Nov. 14, 2014, 1:53 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, line 138
> > <https://reviews.apache.org/r/28017/diff/1/?file=763012#file763012line138>
> >
> > currentTask seems to be the container for sparkWork. Do we need to pass in both of them?
> > BTW, currentTask seems to be a misleading varaible name.
Good point. I always forget this.. Changed the name to "originalTask".
- Chao
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28017/#review61373
-----------------------------------------------------------
On Nov. 14, 2014, 12:03 a.m., Chao Sun wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28017/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2014, 12:03 a.m.)
>
>
> Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
>
>
> Bugs: HIVE-8776
> https://issues.apache.org/jira/browse/HIVE-8776
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables.
> We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 4b9a6cb
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0
>
> Diff: https://reviews.apache.org/r/28017/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chao Sun
>
>
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in
SparkMapJoinResolver [Spark Brach]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28017/#review61373
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/28017/#comment102946>
currentTask seems to be the container for sparkWork. Do we need to pass in both of them?
BTW, currentTask seems to be a misleading varaible name.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
<https://reviews.apache.org/r/28017/#comment102947>
Can we have a local variable for sparkWork.getAllWorkUnsorted()? It's called again later in the code.
- Xuefu Zhang
On Nov. 14, 2014, 12:03 a.m., Chao Sun wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28017/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2014, 12:03 a.m.)
>
>
> Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang.
>
>
> Bugs: HIVE-8776
> https://issues.apache.org/jira/browse/HIVE-8776
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables.
> We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 4b9a6cb
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0
>
> Diff: https://reviews.apache.org/r/28017/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chao Sun
>
>