You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chao Sun <ch...@cloudera.com> on 2014/10/14 23:17:13 UTC

Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

Review request for hive and Xuefu Zhang.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 95d7b59 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out b425c67 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out dc713b3 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out cd8e85e 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 801ac8a 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out b04e55c 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4bde6ea 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out ab2fe84 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 5c1cbc4 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 90bc8ea 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 7964802 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out 31ebbeb 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 0a983d8 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 68b1312 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out f7867ac 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out dbb78a6 
  ql/src/test/results/clientpositive/spark/orc_analyze.q.out a0af7ba 
  ql/src/test/results/clientpositive/spark/parallel.q.out acd418f 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 169d2f1 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 54b8a8a 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 6f8066d 
  ql/src/test/results/clientpositive/spark/union18.q.out 07ea2c5 
  ql/src/test/results/clientpositive/spark/union19.q.out 2fefe8e 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 147f1fe 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out e12943c 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review56647
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97038>

    Please throw IllegalStateException, prefix with "AssertionError " and append work.getClass().getName() to this message


- Brock Noland


On Oct. 14, 2014, 9:17 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 14, 2014, 9:17 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 95d7b59 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out b425c67 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out dc713b3 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out cd8e85e 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 801ac8a 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out b04e55c 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4bde6ea 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out ab2fe84 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 5c1cbc4 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 90bc8ea 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 7964802 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out 31ebbeb 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 0a983d8 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 68b1312 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out f7867ac 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out dbb78a6 
>   ql/src/test/results/clientpositive/spark/orc_analyze.q.out a0af7ba 
>   ql/src/test/results/clientpositive/spark/parallel.q.out acd418f 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 169d2f1 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 54b8a8a 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 6f8066d 
>   ql/src/test/results/clientpositive/spark/union18.q.out 07ea2c5 
>   ql/src/test/results/clientpositive/spark/union19.q.out 2fefe8e 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 147f1fe 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out e12943c 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review56640
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/26706/#comment97030>

    Can we try a generic method so that we only have one method doing cloning for both?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java
<https://reviews.apache.org/r/26706/#comment97031>

    I think input param can be just BytesWritable.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97033>

    I think we should use add() instead.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97035>

    The design doc explicitly specifies that the first clone is handled differently than the rest, but I didn't see such handling here. We may have problem with this implementation.



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/26706/#comment97036>

    Let's not to use * in imports.


- Xuefu Zhang


On Oct. 14, 2014, 9:17 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 14, 2014, 9:17 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 95d7b59 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out b425c67 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out dc713b3 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out cd8e85e 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 801ac8a 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out b04e55c 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4bde6ea 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out ab2fe84 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 5c1cbc4 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 90bc8ea 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 7964802 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out 31ebbeb 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 0a983d8 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 68b1312 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out f7867ac 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out dbb78a6 
>   ql/src/test/results/clientpositive/spark/orc_analyze.q.out a0af7ba 
>   ql/src/test/results/clientpositive/spark/parallel.q.out acd418f 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 169d2f1 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 54b8a8a 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 6f8066d 
>   ql/src/test/results/clientpositive/spark/union18.q.out 07ea2c5 
>   ql/src/test/results/clientpositive/spark/union19.q.out 2fefe8e 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 147f1fe 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out e12943c 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.

> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java, line 64
> > <https://reviews.apache.org/r/26706/diff/4/?file=724853#file724853line64>
> >
> >     Could we reuse this as a utility? I think we have same/similar thing somewhere.

You're right - HiveBaseFunctionResultList has the same method.
I've put it in the SparkUtilities.


> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 250
> > <https://reviews.apache.org/r/26706/diff/4/?file=724854#file724854line250>
> >
> >     Do we need to disconnect it or remove does this automatically?

Yes, remove also remove all edges connected to this node.


> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote:
> > ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q, line 1
> > <https://reviews.apache.org/r/26706/diff/4/?file=724864#file724864line1>
> >
> >     Could we put this test as spark only, as splitting doesn't apply mr or tez? I think we have a dir for spark only tests.

I also wanted to make this as a spark-only test.
But the feature hasn't been implemented yet (I think Szehon is working on it).
I made the file name to start with "spark_" so in future we can move it to spark-only test directory.
But currently, there's no test dir for spark, only result dir.


- Chao


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57286
-----------------------------------------------------------


On Oct. 17, 2014, 9:24 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 17, 2014, 9:24 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
>   ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
>   ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
>   ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
>   ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.

> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote:
> > ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q, line 1
> > <https://reviews.apache.org/r/26706/diff/4/?file=724864#file724864line1>
> >
> >     Could we put this test as spark only, as splitting doesn't apply mr or tez? I think we have a dir for spark only tests.
> 
> Chao Sun wrote:
>     I also wanted to make this as a spark-only test.
>     But the feature hasn't been implemented yet (I think Szehon is working on it).
>     I made the file name to start with "spark_" so in future we can move it to spark-only test directory.
>     But currently, there's no test dir for spark, only result dir.
> 
> Xuefu Zhang wrote:
>     In the case, let rename the test to have a generic name. it's a valid test case for MR also, but also a special case for Spark.

OK, thanks. I've updated the patch accordingly.


- Chao


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57286
-----------------------------------------------------------


On Oct. 19, 2014, 12:46 a.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 19, 2014, 12:46 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
>   ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
>   ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
>   ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
>   ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.

> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote:
> > ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q, line 1
> > <https://reviews.apache.org/r/26706/diff/4/?file=724864#file724864line1>
> >
> >     Could we put this test as spark only, as splitting doesn't apply mr or tez? I think we have a dir for spark only tests.
> 
> Chao Sun wrote:
>     I also wanted to make this as a spark-only test.
>     But the feature hasn't been implemented yet (I think Szehon is working on it).
>     I made the file name to start with "spark_" so in future we can move it to spark-only test directory.
>     But currently, there's no test dir for spark, only result dir.

In the case, let rename the test to have a generic name. it's a valid test case for MR also, but also a special case for Spark.


- Xuefu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57286
-----------------------------------------------------------


On Oct. 19, 2014, 12:46 a.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 19, 2014, 12:46 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
>   ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
>   ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
>   ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
>   ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57286
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/26706/#comment97861>

    Could you remove if this is not applicable?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java
<https://reviews.apache.org/r/26706/#comment97862>

    Could we reuse this as a utility? I think we have same/similar thing somewhere.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97863>

    Let's keep the same style of import organization consistent with the rest of Hive. This is usually on the top, with an abc order.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97864>

    Can we rename this to generate... instead of get... to be more precise?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97866>

    Nit: it might be better if this line comes after the following if statement.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97867>

    If check here is unnecessary.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97857>

    Do we need to disconnect it or remove does this automatically?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/26706/#comment97856>

    Nit: code style.



ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java
<https://reviews.apache.org/r/26706/#comment97858>

    Some java doc would be helpful.



ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q
<https://reviews.apache.org/r/26706/#comment97859>

    Could we put this test as spark only, as splitting doesn't apply mr or tez? I think we have a dir for spark only tests.



ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q
<https://reviews.apache.org/r/26706/#comment97860>

    Nit: trailing space.


- Xuefu Zhang


On Oct. 17, 2014, 9:24 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 17, 2014, 9:24 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
>   ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
>   ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
>   ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
>   ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.

> On Oct. 20, 2014, 9:52 p.m., Xuefu Zhang wrote:
> > itests/src/test/resources/testconfiguration.properties, line 509
> > <https://reviews.apache.org/r/26706/diff/7/?file=726397#file726397line509>
> >
> >     We might need to change this as well.

Can't believe I missed this. Sorry for the sloppyness!


- Chao


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57445
-----------------------------------------------------------


On Oct. 20, 2014, 9:10 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 20, 2014, 9:10 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/multi_insert_mixed.q PRE-CREATION 
>   ql/src/test/results/clientpositive/multi_insert_mixed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 310f2fe 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out e6054c9 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out d0f3e76 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out d40c7bb 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out b4ded62 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out d2529bb 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 7fa6130 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 4a4070b 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 62c179e 
>   ql/src/test/results/clientpositive/spark/input12.q.out a4b7a3c 
>   ql/src/test/results/clientpositive/spark/input13.q.out 5c799dc 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1105ed8 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out 514f54a 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 1b88026 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5b2aa78 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out cbf7204 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 3905d84 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out 0404119 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 903e966 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 730fb4f 
>   ql/src/test/results/clientpositive/spark/multi_insert_mixed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out 1f31f56 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 4ded9d2 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 2b63321 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 16bfac1 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 05d719a 
>   ql/src/test/results/clientpositive/spark/union18.q.out ce3e20c 
>   ql/src/test/results/clientpositive/spark/union19.q.out ac28e36 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 1836150 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 179edd1 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57445
-----------------------------------------------------------



itests/src/test/resources/testconfiguration.properties
<https://reviews.apache.org/r/26706/#comment98143>

    We might need to change this as well.


- Xuefu Zhang


On Oct. 20, 2014, 9:10 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 20, 2014, 9:10 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/multi_insert_mixed.q PRE-CREATION 
>   ql/src/test/results/clientpositive/multi_insert_mixed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 310f2fe 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out e6054c9 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out d0f3e76 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out d40c7bb 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out b4ded62 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out d2529bb 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 7fa6130 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 4a4070b 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 62c179e 
>   ql/src/test/results/clientpositive/spark/input12.q.out a4b7a3c 
>   ql/src/test/results/clientpositive/spark/input13.q.out 5c799dc 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1105ed8 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out 514f54a 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 1b88026 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5b2aa78 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out cbf7204 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 3905d84 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out 0404119 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 903e966 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 730fb4f 
>   ql/src/test/results/clientpositive/spark/multi_insert_mixed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out 1f31f56 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 4ded9d2 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 2b63321 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 16bfac1 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 05d719a 
>   ql/src/test/results/clientpositive/spark/union18.q.out ce3e20c 
>   ql/src/test/results/clientpositive/spark/union19.q.out ac28e36 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 1836150 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 179edd1 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 20, 2014, 10:04 p.m.)


Review request for hive and Xuefu Zhang.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/multi_insert_mixed.q PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_mixed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 310f2fe 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out e6054c9 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out d0f3e76 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out d40c7bb 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out b4ded62 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out d2529bb 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 7fa6130 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 4a4070b 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 62c179e 
  ql/src/test/results/clientpositive/spark/input12.q.out a4b7a3c 
  ql/src/test/results/clientpositive/spark/input13.q.out 5c799dc 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1105ed8 
  ql/src/test/results/clientpositive/spark/input_part2.q.out 514f54a 
  ql/src/test/results/clientpositive/spark/insert1.q.out 1b88026 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5b2aa78 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out cbf7204 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 3905d84 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out 0404119 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 903e966 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 730fb4f 
  ql/src/test/results/clientpositive/spark/multi_insert_mixed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out 1f31f56 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 4ded9d2 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 2b63321 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 16bfac1 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 05d719a 
  ql/src/test/results/clientpositive/spark/union18.q.out ce3e20c 
  ql/src/test/results/clientpositive/spark/union19.q.out ac28e36 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 1836150 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 179edd1 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------

All multi-insertion related results are regenerated, and manually checked against the old results.
Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 20, 2014, 9:10 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Changed test file name and comments, also rebased to the latest update.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/multi_insert_mixed.q PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_mixed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 310f2fe 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out e6054c9 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out d0f3e76 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out d40c7bb 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out b4ded62 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out d2529bb 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 7fa6130 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 4a4070b 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 62c179e 
  ql/src/test/results/clientpositive/spark/input12.q.out a4b7a3c 
  ql/src/test/results/clientpositive/spark/input13.q.out 5c799dc 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1105ed8 
  ql/src/test/results/clientpositive/spark/input_part2.q.out 514f54a 
  ql/src/test/results/clientpositive/spark/insert1.q.out 1b88026 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5b2aa78 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out cbf7204 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 3905d84 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out 0404119 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 903e966 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out 730fb4f 
  ql/src/test/results/clientpositive/spark/multi_insert_mixed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out 1f31f56 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 4ded9d2 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 2b63321 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 16bfac1 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 05d719a 
  ql/src/test/results/clientpositive/spark/union18.q.out ce3e20c 
  ql/src/test/results/clientpositive/spark/union19.q.out ac28e36 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 1836150 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 179edd1 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------

All multi-insertion related results are regenerated, and manually checked against the old results.
Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/#review57412
-----------------------------------------------------------



itests/src/test/resources/testconfiguration.properties
<https://reviews.apache.org/r/26706/#comment98113>

    "split" in the name seems a little confusing. Could we call it multi_insert_mixed.q?



ql/src/test/queries/clientpositive/multi_insert_split_work.q
<https://reviews.apache.org/r/26706/#comment98111>

    could we update the comments here? I guess the test case is special in that some inserts are map only while others invovles a shuffle.


- Xuefu Zhang


On Oct. 20, 2014, 5:38 p.m., Chao Sun wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26706/
> -----------------------------------------------------------
> 
> (Updated Oct. 20, 2014, 5:38 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8436
>     https://issues.apache.org/jira/browse/HIVE-8436
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
> This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
> This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 558dd02 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
>   ql/src/test/queries/clientpositive/multi_insert_split_work.q PRE-CREATION 
>   ql/src/test/results/clientpositive/multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
>   ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
>   ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
>   ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
>   ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
>   ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
>   ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
>   ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
>   ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
>   ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
>   ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
>   ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
>   ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
>   ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
>   ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
>   ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
>   ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
>   ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
>   ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
>   ql/src/test/results/clientpositive/spark/multi_insert_split_work.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
>   ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
>   ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
>   ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
>   ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
>   ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
>   ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
>   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
> 
> Diff: https://reviews.apache.org/r/26706/diff/
> 
> 
> Testing
> -------
> 
> All multi-insertion related results are regenerated, and manually checked against the old results.
> Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 20, 2014, 5:38 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Renamed the test to make it a generic one.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/multi_insert_split_work.q PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------

All multi-insertion related results are regenerated, and manually checked against the old results.
Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 19, 2014, 12:46 a.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Updated the patch according to comments. Thanks Xuefu!


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java c956101 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 3773dcb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
  ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------

All multi-insertion related results are regenerated, and manually checked against the old results.
Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 17, 2014, 9:24 p.m.)


Review request for hive and Xuefu Zhang.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
  ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26706/diff/


Testing (updated)
-------

All multi-insertion related results are regenerated, and manually checked against the old results.
Also I created a new test "spark_multi_insert_spill_work.q" to check splitting won't generate duplicate FSs.


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 17, 2014, 9:22 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Included a qfile result for MR mode.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 558dd02 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 
  ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 17, 2014, 6:04 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Added a test to check that splitting work doesn't create duplicate FSs.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q PRE-CREATION 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------


Thanks,

Chao Sun


Re: Review Request 26706: HIVE-8436 - Modify SparkWork to split works with multiple child works [Spark Branch]

Posted by Chao Sun <ch...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------

(Updated Oct. 16, 2014, 1:25 a.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Addressing the comments. Also, I'm thinking about adding another test for multi-insert in another JIRA, specifically check if the plan after splitting is in the correct shape.


Bugs: HIVE-8436
    https://issues.apache.org/jira/browse/HIVE-8436


Repository: hive-git


Description
-------

Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.
This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.
This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java d7744e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java 1d01040 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java 93940bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 20eb344 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java a62643a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 
  ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 
  ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 
  ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b 
  ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa 
  ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out 399fe41 
  ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 
  ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e 
  ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 
  ql/src/test/results/clientpositive/spark/input13.q.out 260a65a 
  ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 
  ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d 
  ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb 
  ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b 
  ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc 
  ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d 
  ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f 
  ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 
  ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out b07c582 
  ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out fd477ca 
  ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3 
  ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 
  ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 
  ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b 
  ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 
  ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b 

Diff: https://reviews.apache.org/r/26706/diff/


Testing
-------


Thanks,

Chao Sun