You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Robert Surówka (JIRA)" <ji...@apache.org> on 2011/09/14 23:24:09 UTC
[jira] [Created] (HIVE-2446) Introduction of client statistics
publishers possibility
Introduction of client statistics publishers possibility
--------------------------------------------------------
Key: HIVE-2446
URL: https://issues.apache.org/jira/browse/HIVE-2446
Project: Hive
Issue Type: Improvement
Components: Clients, Statistics
Reporter: Robert Surówka
Priority: Minor
The purpose of this change is to allow publication or storage of counters while the job is running.
Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
Made cosmetic changes in two other classes
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105710#comment-13105710 ]
jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/#review1918
-----------------------------------------------------------
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
<https://reviews.apache.org/r/1909/#comment4399>
should -1 be replaced by 0 here?
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
<https://reviews.apache.org/r/1909/#comment4400>
I've thought it over again and feel it's better to use the counter "name" here rather than "display name". Display name has the benefit that the users see the same name at the JT page and the metrics page, but display name could be too long (sometimes may contain special characters) and it could be changed over time. I think "name" should be a better ID here. The only caveat is that the person who's looking at the metrics need to know how it is translated to the display name in JT page.
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
<https://reviews.apache.org/r/1909/#comment4398>
Is this line too long? Hive's line length shouldn't exceed 100 chars.
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java
<https://reviews.apache.org/r/1909/#comment4401>
Rather than passing JobID, it would be more flexible to pass a String converted from jobID. This is the approach that StatsPublisher interface is using.
- Ning
On 2011-09-14 22:19:21, Robert Surówka wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-14 22:19:21)
bq.
bq.
bq. Review request for hive and Ning Zhang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The purpose of this change is to allow publication or storage of counters while the job is running.
bq.
bq. Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
bq.
bq. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
bq.
bq. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
bq.
bq. Made cosmetic changes in two other classes
bq.
bq.
bq. This addresses bug HIVE-2446.
bq. https://issues.apache.org/jira/browse/HIVE-2446
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1170867
bq. trunk/conf/hive-default.xml 1170867
bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1170867
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1170867
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1170867
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Run some random tests, and still running the unit tests.
bq.
bq.
bq. Thanks,
bq.
bq. Robert
bq.
bq.
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104944#comment-13104944 ]
jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/
-----------------------------------------------------------
Review request for hive and Ning Zhang.
Summary
-------
The purpose of this change is to allow publication or storage of counters while the job is running.
Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
Made cosmetic changes in two other classes
This addresses bug HIVE-2446.
https://issues.apache.org/jira/browse/HIVE-2446
Diffs
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1170867
trunk/conf/hive-default.xml 1170867
trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1170867
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1170867
trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1170867
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION
Diff: https://reviews.apache.org/r/1909/diff
Testing
-------
Run some random tests, and still running the unit tests.
Thanks,
Robert
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106625#comment-13106625 ]
jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/#review1935
-----------------------------------------------------------
Ship it!
- Ning
On 2011-09-16 17:35:58, Robert Surówka wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-16 17:35:58)
bq.
bq.
bq. Review request for hive and Ning Zhang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The purpose of this change is to allow publication or storage of counters while the job is running.
bq.
bq. Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
bq.
bq. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
bq.
bq. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
bq.
bq. Made cosmetic changes in one other class
bq.
bq.
bq. This addresses bug HIVE-2446.
bq. https://issues.apache.org/jira/browse/HIVE-2446
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171283
bq. trunk/conf/hive-default.xml 1171283
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171283
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1171283
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Run some random tests, and still running the unit tests.
bq.
bq.
bq. Thanks,
bq.
bq. Robert
bq.
bq.
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.1.patch
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.3.patch
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.2.patch
The final version of this patch, that was reviewed
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.1.patch
solution
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107098#comment-13107098 ]
Hudson commented on HIVE-2446:
------------------------------
Integrated in Hive-trunk-h0.21 #960 (See [https://builds.apache.org/job/Hive-trunk-h0.21/960/])
HIVE-2446. Introduction of client statistics publishers possibility (Robert Surówka via Ning Zhang)
nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1171917
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130109#comment-13130109 ]
Hudson commented on HIVE-2446:
------------------------------
Integrated in Hive-trunk-h0.21 #1019 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1019/])
HIVE-2487: Bug from HIVE-2446, the code that calls client stats publishers should be inside the while loop (Robert Surówka via He Yongqiang)
heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185767
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (Reopened JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka reopened HIVE-2446:
----------------------------------
Client stat's publishers run() methods are called only once per HadoopJobExecHelper's progress method being called, whereas they should be called inside of while (!rj.isComplete()) {} loop.
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-2446:
--------------------------------
Assignee: Robert Surówka
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106633#comment-13106633 ]
Ning Zhang commented on HIVE-2446:
----------------------------------
+1. Will commit if tests pass.
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Release Note: See https://reviews.apache.org/r/2182/diff/#index_header
Hadoop Flags: (was: Reviewed)
Status: Patch Available (was: Reopened)
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106620#comment-13106620 ]
jiraposter@reviews.apache.org commented on HIVE-2446:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1909/
-----------------------------------------------------------
(Updated 2011-09-16 17:35:58.909415)
Review request for hive and Ning Zhang.
Changes
-------
Changed interface of ClientStatsPublisher, made instantiation of Client Stats Publishers more efficient, this diff no longer removes some unused imports
Summary (updated)
-------
The purpose of this change is to allow publication or storage of counters while the job is running.
Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
Made cosmetic changes in one other class
This addresses bug HIVE-2446.
https://issues.apache.org/jira/browse/HIVE-2446
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171283
trunk/conf/hive-default.xml 1171283
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171283
trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1171283
trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION
Diff: https://reviews.apache.org/r/1909/diff
Testing
-------
Run some random tests, and still running the unit tests.
Thanks,
Robert
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Status: Patch Available (was: Open)
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Priority: Minor
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "He Yongqiang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-2446:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
move the fix to a new jira ticket
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Status: Open (was: Patch Available)
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135498#comment-13135498 ]
Hudson commented on HIVE-2446:
------------------------------
Integrated in Hive-trunk-h0.21 #1033 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1033/])
HIVE-2446. bug (next one) - If constructor of ClientStatsPublisher throws runtime exception it will be propagated to HadoopJobExecHelper progress method and beyond, whereas it shouldnot
nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188796
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang resolved HIVE-2446.
------------------------------
Resolution: Fixed
Fix Version/s: 0.8.0
Hadoop Flags: [Reviewed]
Committed. Thanks Robert!
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.3.patch
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch, HIVE-2446.2.patch, HIVE-2446.3.patch, HIVE-2446.3.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2446) Introduction of client statistics
publishers possibility
Posted by "Robert Surówka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2446:
---------------------------------
Attachment: HIVE-2446.2.patch
> Introduction of client statistics publishers possibility
> --------------------------------------------------------
>
> Key: HIVE-2446
> URL: https://issues.apache.org/jira/browse/HIVE-2446
> Project: Hive
> Issue Type: Improvement
> Components: Clients, Statistics
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The purpose of this change is to allow publication or storage of counters while the job is running.
> Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.
> Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.
> Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.
> Made cosmetic changes in two other classes
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira