You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Brock Noland <br...@cloudera.com> on 2014/08/21 02:26:39 UTC

Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/
-----------------------------------------------------------

Review request for hive.


Repository: hive-git


Description
-------

Enable tests


Diffs
-----

  itests/src/test/resources/testconfiguration.properties ecb8b74 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
  ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24918/diff/


Testing
-------

Verified output vs MR


Thanks,

Brock Noland


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Brock Noland <br...@cloudera.com>.

> On Aug. 21, 2014, 5:29 p.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 195
> > <https://reviews.apache.org/r/24918/diff/1/?file=665809#file665809line195>
> >
> >     I'm just curious, was this to fix a test?
> >     
> >     Also is it necessary to catch and wrap in RuntimeException here?

Good point. In the next rev I will wrap, but with a better exception and error message.


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------


On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2014, 12:26 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Enable tests
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties ecb8b74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24918/diff/
> 
> 
> Testing
> -------
> 
> Verified output vs MR
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Brock Noland <br...@cloudera.com>.

> On Aug. 21, 2014, 5:29 p.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 195
> > <https://reviews.apache.org/r/24918/diff/1/?file=665809#file665809line195>
> >
> >     I'm just curious, was this to fix a test?
> >     
> >     Also is it necessary to catch and wrap in RuntimeException here?
> 
> Brock Noland wrote:
>     Good point. In the next rev I will wrap, but with a better exception and error message.

And yes this was to fix bucketed tables.


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------


On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2014, 12:26 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Enable tests
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties ecb8b74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24918/diff/
> 
> 
> Testing
> -------
> 
> Verified output vs MR
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------


Hey Brock, thanks, just some pretty minor comments below.


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/24918/#comment89218>

    I'm just curious, was this to fix a test?
    
    Also is it necessary to catch and wrap in RuntimeException here?



ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/24918/#comment89216>

    This is a bit strange, as the resolve() method returns null.  Not sure if we should assign to a variable as of now?


- Szehon Ho


On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2014, 12:26 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Enable tests
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties ecb8b74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24918/diff/
> 
> 
> Testing
> -------
> 
> Verified output vs MR
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/
-----------------------------------------------------------

(Updated Aug. 22, 2014, 11:41 p.m.)


Review request for hive.


Repository: hive-git


Description
-------

Enable tests


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties de4a36e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 04dc382 
  ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24918/diff/


Testing
-------

Verified output vs MR


Thanks,

Brock Noland


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Brock Noland <br...@cloudera.com>.

> On Aug. 21, 2014, 8:48 p.m., Venki Korukanti wrote:
> > ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out, line 29
> > <https://reviews.apache.org/r/24918/diff/1/?file=665818#file665818line29>
> >
> >     I am working on enabling .q files listed in HIVE-7792, I see several tests just having this error. Is it ok to check .out files containing these errors?

Yes, this is just because we have not implemented the stats functionality...


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51220
-----------------------------------------------------------


On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2014, 12:26 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Enable tests
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties ecb8b74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24918/diff/
> 
> 
> Testing
> -------
> 
> Verified output vs MR
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1) [Sparch Branch]

Posted by Venki Korukanti <ve...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51220
-----------------------------------------------------------



ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out
<https://reviews.apache.org/r/24918/#comment89290>

    I am working on enabling .q files listed in HIVE-7792, I see several tests just having this error. Is it ok to check .out files containing these errors?


- Venki Korukanti


On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2014, 12:26 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Enable tests
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties ecb8b74 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24918/diff/
> 
> 
> Testing
> -------
> 
> Verified output vs MR
> 
> 
> Thanks,
> 
> Brock Noland
> 
>