You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Vikram Dixit Kumaraswamy <vi...@gmail.com> on 2013/10/15 23:50:19 UTC

Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/
-----------------------------------------------------------

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5543
    https://issues.apache.org/jira/browse/HIVE-5543


Repository: hive-git


Description
-------

Running the mini tez cluster for tez unit tests.


Diffs
-----

  build-common.xml ee6a3ce 
  build.properties 3dbd33f 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 0558048 
  hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
  hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
  ql/build.xml be8e10f 
  ql/ivy.xml d0159d9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
  ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
  ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
  ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
  ql/src/test/templates/TestCliDriver.vm f2623f8 
  ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
  ql/src/test/templates/TestParse.vm e28e901 
  ql/src/test/templates/TestParseNegative.vm 1aec083 
  shims/build.xml cbfba5a 
  shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
  shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
  shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 

Diff: https://reviews.apache.org/r/14651/diff/


Testing
-------

Added new tests for running the tez mini cluster and a new test target.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

Posted by Brock Noland <br...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/#review27489
-----------------------------------------------------------



ql/build.xml
<https://reviews.apache.org/r/14651/#comment53417>

    hey guys, I don't plan on doing a review of this, but it'd be great if we could trim this trailing whitespace.
    
    Sorry for being a PIA :)


- Brock Noland


On Oct. 15, 2013, 9:50 p.m., Vikram Dixit Kumaraswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14651/
> -----------------------------------------------------------
> 
> (Updated Oct. 15, 2013, 9:50 p.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-5543
>     https://issues.apache.org/jira/browse/HIVE-5543
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Running the mini tez cluster for tez unit tests.
> 
> 
> Diffs
> -----
> 
>   build-common.xml ee6a3ce 
>   build.properties 3dbd33f 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 0558048 
>   hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
>   hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
>   ql/build.xml be8e10f 
>   ql/ivy.xml d0159d9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
>   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
>   ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
>   ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
>   ql/src/test/templates/TestCliDriver.vm f2623f8 
>   ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
>   ql/src/test/templates/TestParse.vm e28e901 
>   ql/src/test/templates/TestParseNegative.vm 1aec083 
>   shims/build.xml cbfba5a 
>   shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
>   shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
>   shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14651/diff/
> 
> 
> Testing
> -------
> 
> Added new tests for running the tez mini cluster and a new test target.
> 
> 
> Thanks,
> 
> Vikram Dixit Kumaraswamy
> 
>

Re: Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

Posted by Gunther Hagleitner <gh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/#review27481
-----------------------------------------------------------



ql/build.xml
<https://reviews.apache.org/r/14651/#comment53401>

    not sure what logDir does, but do we need a special one for that? are the files all going to be prefixed with TestMiniTez?



ql/ivy.xml
<https://reviews.apache.org/r/14651/#comment53402>

    is runtime internals where they expose the mini tez cluster?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
<https://reviews.apache.org/r/14651/#comment53403>

    isn't this redundant with the code below? both will merge in hiveConf right?



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/14651/#comment53404>

    this is a little confusing. how about a mode enum? local, mr, tez or something.



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/14651/#comment53408>

    It seems you only need a MiniMrShim. It would definitely be much less code esp in build xml stuff if you just create that class in test/org/apache/hive/ql/ and plop it in here. Thoughts?
    
    



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/14651/#comment53406>

    this seems just more verbose...



shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
<https://reviews.apache.org/r/14651/#comment53407>

    again just seems more verbose...


- Gunther Hagleitner


On Oct. 15, 2013, 9:50 p.m., Vikram Dixit Kumaraswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14651/
> -----------------------------------------------------------
> 
> (Updated Oct. 15, 2013, 9:50 p.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-5543
>     https://issues.apache.org/jira/browse/HIVE-5543
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Running the mini tez cluster for tez unit tests.
> 
> 
> Diffs
> -----
> 
>   build-common.xml ee6a3ce 
>   build.properties 3dbd33f 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 0558048 
>   hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
>   hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
>   ql/build.xml be8e10f 
>   ql/ivy.xml d0159d9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
>   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
>   ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
>   ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
>   ql/src/test/templates/TestCliDriver.vm f2623f8 
>   ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
>   ql/src/test/templates/TestParse.vm e28e901 
>   ql/src/test/templates/TestParseNegative.vm 1aec083 
>   shims/build.xml cbfba5a 
>   shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
>   shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
>   shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14651/diff/
> 
> 
> Testing
> -------
> 
> Added new tests for running the tez mini cluster and a new test target.
> 
> 
> Thanks,
> 
> Vikram Dixit Kumaraswamy
> 
>

Re: Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

Posted by Brock Noland <br...@cloudera.com>.


> On Oct. 29, 2013, 9:48 a.m., Vikram Dixit Kumaraswamy wrote:
> > ql/build.xml, line 138
> > <https://reviews.apache.org/r/14651/diff/1/?file=364830#file364830line138>
> >
> >     I will do it. Just to understand though, does this affect mavenization or is an aesthetic concern?

I don't see how whitespace could affect mavenization? :)

It's an aesthetic concern, but something that should be addressed. We should not commit trailing whitespace whenever possible, that is why it shows up in red.

Thanks guys!


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/#review27559
-----------------------------------------------------------


On Oct. 15, 2013, 9:50 p.m., Vikram Dixit Kumaraswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14651/
> -----------------------------------------------------------
> 
> (Updated Oct. 15, 2013, 9:50 p.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-5543
>     https://issues.apache.org/jira/browse/HIVE-5543
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Running the mini tez cluster for tez unit tests.
> 
> 
> Diffs
> -----
> 
>   build-common.xml ee6a3ce 
>   build.properties 3dbd33f 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 0558048 
>   hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
>   hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
>   ql/build.xml be8e10f 
>   ql/ivy.xml d0159d9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
>   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
>   ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
>   ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
>   ql/src/test/templates/TestCliDriver.vm f2623f8 
>   ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
>   ql/src/test/templates/TestParse.vm e28e901 
>   ql/src/test/templates/TestParseNegative.vm 1aec083 
>   shims/build.xml cbfba5a 
>   shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
>   shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
>   shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14651/diff/
> 
> 
> Testing
> -------
> 
> Added new tests for running the tez mini cluster and a new test target.
> 
> 
> Thanks,
> 
> Vikram Dixit Kumaraswamy
> 
>

Re: Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

Posted by Vikram Dixit Kumaraswamy <vi...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/#review27559
-----------------------------------------------------------



ql/build.xml
<https://reviews.apache.org/r/14651/#comment53488>

    



ql/build.xml
<https://reviews.apache.org/r/14651/#comment53489>

    I will do it. Just to understand though, does this affect mavenization or is an aesthetic concern?



ql/build.xml
<https://reviews.apache.org/r/14651/#comment53490>

    log dir is the directory where results are put temporarily before being compared against the results directory. It will not prefix files with TestMiniTez.



ql/ivy.xml
<https://reviews.apache.org/r/14651/#comment53487>

    No. But they use classes in this jar to execute. Since in the case of mini tez cluster, the jars are not available in TEZ_HOME etc., there is no access to this jar unless it is put on the classpath. Hence the ivy dependency to pull in this jar.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
<https://reviews.apache.org/r/14651/#comment53486>

    The for loop below only updates configurations that are null and respectively non-null within the hive conf. This was wiping out some of the configurations because of defaults coming into play. The new update fixes this as this is how the api used in the latest code.



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/14651/#comment53491>

    Fixed.



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
<https://reviews.apache.org/r/14651/#comment53492>

    Fixed



shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
<https://reviews.apache.org/r/14651/#comment53493>

    Fixed.



shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
<https://reviews.apache.org/r/14651/#comment53494>

    Fixed.


- Vikram Dixit Kumaraswamy


On Oct. 15, 2013, 9:50 p.m., Vikram Dixit Kumaraswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14651/
> -----------------------------------------------------------
> 
> (Updated Oct. 15, 2013, 9:50 p.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-5543
>     https://issues.apache.org/jira/browse/HIVE-5543
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Running the mini tez cluster for tez unit tests.
> 
> 
> Diffs
> -----
> 
>   build-common.xml ee6a3ce 
>   build.properties 3dbd33f 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 0558048 
>   hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
>   hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
>   ql/build.xml be8e10f 
>   ql/ivy.xml d0159d9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
>   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
>   ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
>   ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
>   ql/src/test/templates/TestCliDriver.vm f2623f8 
>   ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
>   ql/src/test/templates/TestParse.vm e28e901 
>   ql/src/test/templates/TestParseNegative.vm 1aec083 
>   shims/build.xml cbfba5a 
>   shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
>   shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
>   shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14651/diff/
> 
> 
> Testing
> -------
> 
> Added new tests for running the tez mini cluster and a new test target.
> 
> 
> Thanks,
> 
> Vikram Dixit Kumaraswamy
> 
>