You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Szehon Ho <sz...@cloudera.com> on 2015/11/14 02:10:50 UTC
Review Request 40318: HIVE-12271 : Add metrics around HS2 query
execution and job submission for Hive
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40318/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-12271
https://issues.apache.org/jira/browse/HIVE-12271
Repository: hive-git
Description
-------
Add following metrics:
1. Size of async thread pool and queue. (in SessionManager)
2. Number/Duration of all HS2 operations in each state, and count of what state they finished (in Operation)
3. Number/Duration of all calls recorded by PerfLogger (in PerfLogger)
In the metrics, cleaned up the definition of MetricsScope, extended it to also record number of active calls instead of just time. MetricsScope are stored by framework itself for metastore/PerfLogger cases, and kept by the code in Operation (due to nature of operations being run on async threads).
Minor note: removed a PerfLogBegin call for task with specific task-id. It is missing PerfLogEnd so never recorded anyway, and also seemed not useful as all other PerfLogger calls are logged with same method name and this one was the outlyer.
Diffs
-----
common/pom.xml 2292fdf
common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 52d99e4
common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 49b2b32
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java a5aa995
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsScope.java PRE-CREATION
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java 3db26af
common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 548d7db
common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java PRE-CREATION
common/src/test/org/apache/hadoop/hive/common/metrics/TestLegacyMetrics.java c3e8282
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java a3aa549
itests/hive-unit/pom.xml c202001
itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java PRE-CREATION
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3c40d6e
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 93c7a54
service/pom.xml 22234d9
service/src/java/org/apache/hive/service/cli/operation/Operation.java 4ca0561
service/src/java/org/apache/hive/service/cli/session/SessionManager.java a9b4334
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java PRE-CREATION
Diff: https://reviews.apache.org/r/40318/diff/
Testing
-------
Added two unit tests: TestHs2Metrics, TestSessionManagerMetrics, test all of these metric categories in in-flight and post-operation metric counts.
Thanks,
Szehon Ho
Re: Review Request 40318: HIVE-12271 : Add metrics around HS2 query
execution and job submission for Hive
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40318/
-----------------------------------------------------------
(Updated Nov. 16, 2015, 7:21 p.m.)
Review request for hive.
Changes
-------
Rebase.
Bugs: HIVE-12271
https://issues.apache.org/jira/browse/HIVE-12271
Repository: hive-git
Description
-------
Add following metrics:
1. Size of async thread pool and queue. (in SessionManager)
2. Number/Duration of all HS2 operations in each state, and count of what state they finished (in Operation)
3. Number/Duration of all calls recorded by PerfLogger (in PerfLogger)
In the metrics, cleaned up the definition of MetricsScope, extended it to also record number of active calls instead of just time. MetricsScope are stored by framework itself for metastore/PerfLogger cases, and kept by the code in Operation (due to nature of operations being run on async threads).
Minor note: removed a PerfLogBegin call for task with specific task-id. It is missing PerfLogEnd so never recorded anyway, and also seemed not useful as all other PerfLogger calls are logged with same method name and this one was the outlyer.
Diffs (updated)
-----
common/pom.xml 2292fdf
common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 52d99e4
common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 49b2b32
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java a5aa995
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsScope.java PRE-CREATION
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java 3db26af
common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 548d7db
common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java PRE-CREATION
common/src/test/org/apache/hadoop/hive/common/metrics/TestLegacyMetrics.java c3e8282
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java a3aa549
itests/hive-unit/pom.xml c202001
itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java PRE-CREATION
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3c40d6e
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9a5495b
service/pom.xml 22234d9
service/src/java/org/apache/hive/service/cli/operation/Operation.java a851936
service/src/java/org/apache/hive/service/cli/session/SessionManager.java a9b4334
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java PRE-CREATION
Diff: https://reviews.apache.org/r/40318/diff/
Testing
-------
Added two unit tests: TestHs2Metrics, TestSessionManagerMetrics, test all of these metric categories in in-flight and post-operation metric counts.
Thanks,
Szehon Ho