You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Brock Noland <br...@cloudera.com> on 2014/08/21 02:26:39 UTC
Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
Enable tests
Diffs
-----
itests/src/test/resources/testconfiguration.properties ecb8b74
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/24918/diff/
Testing
-------
Verified output vs MR
Thanks,
Brock Noland
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 21, 2014, 5:29 p.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 195
> > <https://reviews.apache.org/r/24918/diff/1/?file=665809#file665809line195>
> >
> > I'm just curious, was this to fix a test?
> >
> > Also is it necessary to catch and wrap in RuntimeException here?
Good point. In the next rev I will wrap, but with a better exception and error message.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------
On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2014, 12:26 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Enable tests
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties ecb8b74
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
> ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24918/diff/
>
>
> Testing
> -------
>
> Verified output vs MR
>
>
> Thanks,
>
> Brock Noland
>
>
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 21, 2014, 5:29 p.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 195
> > <https://reviews.apache.org/r/24918/diff/1/?file=665809#file665809line195>
> >
> > I'm just curious, was this to fix a test?
> >
> > Also is it necessary to catch and wrap in RuntimeException here?
>
> Brock Noland wrote:
> Good point. In the next rev I will wrap, but with a better exception and error message.
And yes this was to fix bucketed tables.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------
On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2014, 12:26 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Enable tests
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties ecb8b74
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
> ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24918/diff/
>
>
> Testing
> -------
>
> Verified output vs MR
>
>
> Thanks,
>
> Brock Noland
>
>
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51187
-----------------------------------------------------------
Hey Brock, thanks, just some pretty minor comments below.
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
<https://reviews.apache.org/r/24918/#comment89218>
I'm just curious, was this to fix a test?
Also is it necessary to catch and wrap in RuntimeException here?
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java
<https://reviews.apache.org/r/24918/#comment89216>
This is a bit strange, as the resolve() method returns null. Not sure if we should assign to a variable as of now?
- Szehon Ho
On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2014, 12:26 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Enable tests
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties ecb8b74
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
> ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24918/diff/
>
>
> Testing
> -------
>
> Verified output vs MR
>
>
> Thanks,
>
> Brock Noland
>
>
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/
-----------------------------------------------------------
(Updated Aug. 22, 2014, 11:41 p.m.)
Review request for hive.
Repository: hive-git
Description
-------
Enable tests
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties de4a36e
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 04dc382
ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/24918/diff/
Testing
-------
Verified output vs MR
Thanks,
Brock Noland
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Brock Noland <br...@cloudera.com>.
> On Aug. 21, 2014, 8:48 p.m., Venki Korukanti wrote:
> > ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out, line 29
> > <https://reviews.apache.org/r/24918/diff/1/?file=665818#file665818line29>
> >
> > I am working on enabling .q files listed in HIVE-7792, I see several tests just having this error. Is it ok to check .out files containing these errors?
Yes, this is just because we have not implemented the stats functionality...
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51220
-----------------------------------------------------------
On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2014, 12:26 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Enable tests
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties ecb8b74
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
> ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24918/diff/
>
>
> Testing
> -------
>
> Verified output vs MR
>
>
> Thanks,
>
> Brock Noland
>
>
Re: Review Request 24918: HIVE-7791 - Enable tests on Spark branch (1)
[Sparch Branch]
Posted by Venki Korukanti <ve...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24918/#review51220
-----------------------------------------------------------
ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out
<https://reviews.apache.org/r/24918/#comment89290>
I am working on enabling .q files listed in HIVE-7792, I see several tests just having this error. Is it ok to check .out files containing these errors?
- Venki Korukanti
On Aug. 21, 2014, 12:26 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24918/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2014, 12:26 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Enable tests
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties ecb8b74
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java d16f1be
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java dc621cf
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 026f4e0
> ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/alter_merge_stats_orc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/count.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/create_merge_compressed.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ctas.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/custom_input_output_format.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/disable_merge_for_bucketing.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24918/diff/
>
>
> Testing
> -------
>
> Verified output vs MR
>
>
> Thanks,
>
> Brock Noland
>
>