You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chao Sun <ch...@cloudera.com> on 2014/09/27 23:16:43 UTC

Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/
-----------------------------------------------------------

Review request for hive and Xuefu Zhang.


Bugs: HIVE-8278
    https://issues.apache.org/jira/browse/HIVE-8278


Repository: hive-git


Description
-------

HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 

Diff: https://reviews.apache.org/r/26120/diff/


Testing
-------


Thanks,

Chao Sun

Re: Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.

> On Sept. 29, 2014, 11:34 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java, line 34
> > <https://reviews.apache.org/r/26120/diff/2/?file=708344#file708344line34>
> >
> >     Do we need rootTrans?

I thought rootTrans is just the same as the keyset of mapInputs, so I removed it. Correct me if I'm wrong.


> On Sept. 29, 2014, 11:34 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java, line 72
> > <https://reviews.apache.org/r/26120/diff/2/?file=708344#file708344line72>
> >
> >     So we are not do union of those any more?

I forgot ... now fixed.


- Chao


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/#review54904
-----------------------------------------------------------


On Sept. 29, 2014, 8:16 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26120/
> -----------------------------------------------------------
> 
> (Updated Sept. 29, 2014, 8:16 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8278
>     https://issues.apache.org/jira/browse/HIVE-8278
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 
> 
> Diff: https://reviews.apache.org/r/26120/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chao Sun
> 
>

Re: Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/#review54904
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java
<https://reviews.apache.org/r/26120/#comment95236>

    Do we need rootTrans?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java
<https://reviews.apache.org/r/26120/#comment95254>

    So we are not do union of those any more?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java
<https://reviews.apache.org/r/26120/#comment95252>

    Can we make this private? Also, some comments would be good (similar to getAllWorks() in SparkWork).


- Xuefu Zhang


On Sept. 29, 2014, 8:16 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26120/
> -----------------------------------------------------------
> 
> (Updated Sept. 29, 2014, 8:16 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8278
>     https://issues.apache.org/jira/browse/HIVE-8278
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 
> 
> Diff: https://reviews.apache.org/r/26120/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chao Sun
> 
>

Re: Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/#review54941
-----------------------------------------------------------

Ship it!


Ship It!

- Xuefu Zhang


On Sept. 30, 2014, 12:06 a.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26120/
> -----------------------------------------------------------
> 
> (Updated Sept. 30, 2014, 12:06 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8278
>     https://issues.apache.org/jira/browse/HIVE-8278
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 
> 
> Diff: https://reviews.apache.org/r/26120/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chao Sun
> 
>

Re: Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/
-----------------------------------------------------------

(Updated Sept. 30, 2014, 12:06 a.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Thanks Xuefu for the comments. Now a updated patch.


Bugs: HIVE-8278
    https://issues.apache.org/jira/browse/HIVE-8278


Repository: hive-git


Description
-------

HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 

Diff: https://reviews.apache.org/r/26120/diff/


Testing
-------


Thanks,

Chao Sun

Re: Review Request 26120: HIVE-8278 - Restoring a graph representation of SparkPlan [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26120/
-----------------------------------------------------------

(Updated Sept. 29, 2014, 8:16 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Updated according to Xuefu's suggestions.


Bugs: HIVE-8278
    https://issues.apache.org/jira/browse/HIVE-8278


Repository: hive-git


Description
-------

HIVE-8249 greatly simply file the SparkPlan model and the SparkPlanGenerator logic. As a side effect, however, a visual representation of SparkPlan got lost. Such representation is helpful for debugging and performance profiling. In addition, it would be also good to separate plan generation and plan execution.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java f8b3283 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 15af0f9 

Diff: https://reviews.apache.org/r/26120/diff/


Testing
-------


Thanks,

Chao Sun