You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by chengxiang li <ch...@intel.com> on 2014/10/17 09:47:39 UTC
Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-7709
https://issues.apache.org/jira/browse/HIVE-7709
Repository: hive-git
Description
-------
Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 20ea977
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
Diff: https://reviews.apache.org/r/26867/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/
-----------------------------------------------------------
(Updated Oct. 21, 2014, 6:49 a.m.)
Review request for hive.
Bugs: HIVE-7709
https://issues.apache.org/jira/browse/HIVE-7709
Repository: hive-git
Description
-------
Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 02ecc92
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 82c6161
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java e67210f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
Diff: https://reviews.apache.org/r/26867/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
> On 十月 21, 2014, 5:11 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java, line 54
> > <https://reviews.apache.org/r/26867/diff/2/?file=725682#file725682line54>
> >
> > I'm wondering why we don't need this line. It seems that counter stats publisher (CounterStatsPublisher class) is using this to publish statistics.
Temporarily I remove this line, as Hive would enable table statistics collection if reporter is not null, which may lead to qtest failed as extra exception msg would be print out. I would enable table statistic collection based on Counter later, and i would add this line back at that time.
- chengxiang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/#review57548
-----------------------------------------------------------
On 十月 20, 2014, 9:11 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26867/
> -----------------------------------------------------------
>
> (Updated 十月 20, 2014, 9:11 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7709
> https://issues.apache.org/jira/browse/HIVE-7709
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java e67210f
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
>
> Diff: https://reviews.apache.org/r/26867/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/#review57548
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
<https://reviews.apache.org/r/26867/#comment98322>
I'm wondering why we don't need this line. It seems that counter stats publisher (CounterStatsPublisher class) is using this to publish statistics.
- Xuefu Zhang
On Oct. 20, 2014, 9:11 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26867/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 9:11 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-7709
> https://issues.apache.org/jira/browse/HIVE-7709
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java e67210f
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
>
> Diff: https://reviews.apache.org/r/26867/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 9:11 a.m.)
Review request for hive.
Bugs: HIVE-7709
https://issues.apache.org/jira/browse/HIVE-7709
Repository: hive-git
Description
-------
Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java e67210f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
Diff: https://reviews.apache.org/r/26867/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 26867: HIVE-7709 Create SparkReporter[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26867/
-----------------------------------------------------------
(Updated Oct. 17, 2014, 7:48 a.m.)
Review request for hive.
Bugs: HIVE-7709
https://issues.apache.org/jira/browse/HIVE-7709
Repository: hive-git
Description
-------
Hive operators use Reporter to collect global information, with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 3fd8e47
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 7cfd43d
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 5153885
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java 39af1d1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 20ea977
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 126cb9f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReporter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/counter/SparkCounters.java 3c7eb99
Diff: https://reviews.apache.org/r/26867/diff/
Testing
-------
Thanks,
chengxiang li