You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Barna Zsombor Klara <zs...@cloudera.com> on 2016/08/18 00:24:58 UTC
Review Request 51193: HIVE-14358: Add metrics for number of queries
executed for each execution engine (mr, spark, tez)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51193/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark, tez)
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java 9dc96f9c6412720a891b5c55e2074049c893d780
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 183ed829ef1742e48539f8928293d56b77bc43c8
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java eeaa54320ffaa7ba5d6ebece80a0cb4aadc1dada
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java ce1106d91db9ef75e7b425d5950f888bacbfb3e5
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ac922ce486babe042984d87a7f7442cbfc11484f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 0b494aa5548f8e6ae76e2d0eea9a7afb33961f97
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25c4514b34fb2ed4fc8b1238059bd9dc29d2741b
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 53672a9783b4d13c5eed4ef01f5c16af568a0a41
Diff: https://reviews.apache.org/r/51193/diff/
Testing
-------
Ran the new unit tests in the ql project, everything was green.
Checked that the metrics for map reduce and spark tasks were appearing and being incremented correctly using JMX.
Map reduce tasks were being created by a simple select statement containing a join.
Spark tasks were being created by the same query with the spark execution engine being used.
The metrics were correct across several beeline connections, and were reset once the HiveServer2 was restarted.
The metric collection can be turned on/off using the configuration variable "hive.server2.metrics.enabled". No errors/exceptions encountered when the metrics were disabled.
NB only the root tasks are incrementing the counter since the original jira was about counting the number of queries issued against each exeution engine, so a complex query resulting in more than one task should only count as one as per my understanding.
Thanks,
Barna Zsombor Klara
Re: Review Request 51193: HIVE-14358: Add metrics for number of
queries executed for each execution engine (mr, spark, tez)
Posted by Barna Zsombor Klara <zs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51193/
-----------------------------------------------------------
(Updated Aug. 18, 2016, 6:35 p.m.)
Review request for hive.
Changes
-------
As per Peter's comments, I have changed the import statements. Static imports have been moved to the beginning, imported static methods/classes are enumerated instead of using the wildcard.
Repository: hive-git
Description
-------
HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark, tez)
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java 9dc96f9c6412720a891b5c55e2074049c893d780
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 183ed829ef1742e48539f8928293d56b77bc43c8
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java eeaa54320ffaa7ba5d6ebece80a0cb4aadc1dada
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java ce1106d91db9ef75e7b425d5950f888bacbfb3e5
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ac922ce486babe042984d87a7f7442cbfc11484f
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 0b494aa5548f8e6ae76e2d0eea9a7afb33961f97
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25c4514b34fb2ed4fc8b1238059bd9dc29d2741b
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 53672a9783b4d13c5eed4ef01f5c16af568a0a41
Diff: https://reviews.apache.org/r/51193/diff/
Testing
-------
Ran the new unit tests in the ql project, everything was green.
Checked that the metrics for map reduce and spark tasks were appearing and being incremented correctly using JMX.
Map reduce tasks were being created by a simple select statement containing a join.
Spark tasks were being created by the same query with the spark execution engine being used.
The metrics were correct across several beeline connections, and were reset once the HiveServer2 was restarted.
The metric collection can be turned on/off using the configuration variable "hive.server2.metrics.enabled". No errors/exceptions encountered when the metrics were disabled.
NB only the root tasks are incrementing the counter since the original jira was about counting the number of queries issued against each exeution engine, so a complex query resulting in more than one task should only count as one as per my understanding.
Thanks,
Barna Zsombor Klara
Re: Review Request 51193: HIVE-14358: Add metrics for number of
queries executed for each execution engine (mr, spark, tez)
Posted by Peter Vary <pv...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51193/#review146091
-----------------------------------------------------------
Nice clean patch, only find nits.
Maybe who understands the Hive internals better, might find something, but I do not think so.
Thanks,
Peter
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java (line 28)
<https://reviews.apache.org/r/51193/#comment212472>
nit: I think imports should be specific as per coding guideline
ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java (line 27)
<https://reviews.apache.org/r/51193/#comment212470>
nit: I think imports should be specific as per coding guideline
ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java (line 27)
<https://reviews.apache.org/r/51193/#comment212469>
nit: I think imports should be specific as per coding guideline
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java (lines 43 - 44)
<https://reviews.apache.org/r/51193/#comment212468>
nit: I think imports should be specific as per coding guideline
- Peter Vary
On Aug. 18, 2016, 12:24 a.m., Barna Zsombor Klara wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51193/
> -----------------------------------------------------------
>
> (Updated Aug. 18, 2016, 12:24 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-14358: Add metrics for number of queries executed for each execution engine (mr, spark, tez)
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java 9dc96f9c6412720a891b5c55e2074049c893d780
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 183ed829ef1742e48539f8928293d56b77bc43c8
> ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java eeaa54320ffaa7ba5d6ebece80a0cb4aadc1dada
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java ce1106d91db9ef75e7b425d5950f888bacbfb3e5
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ac922ce486babe042984d87a7f7442cbfc11484f
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 0b494aa5548f8e6ae76e2d0eea9a7afb33961f97
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25c4514b34fb2ed4fc8b1238059bd9dc29d2741b
> ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapRedTask.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/mr/TestMapredLocalTask.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkTask.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 53672a9783b4d13c5eed4ef01f5c16af568a0a41
>
> Diff: https://reviews.apache.org/r/51193/diff/
>
>
> Testing
> -------
>
> Ran the new unit tests in the ql project, everything was green.
> Checked that the metrics for map reduce and spark tasks were appearing and being incremented correctly using JMX.
> Map reduce tasks were being created by a simple select statement containing a join.
> Spark tasks were being created by the same query with the spark execution engine being used.
> The metrics were correct across several beeline connections, and were reset once the HiveServer2 was restarted.
> The metric collection can be turned on/off using the configuration variable "hive.server2.metrics.enabled". No errors/exceptions encountered when the metrics were disabled.
>
> NB only the root tasks are incrementing the counter since the original jira was about counting the number of queries issued against each exeution engine, so a complex query resulting in more than one task should only count as one as per my understanding.
>
>
> Thanks,
>
> Barna Zsombor Klara
>
>