You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gary Helmling (JIRA)" <ji...@apache.org> on 2011/03/09 20:24:01 UTC
[jira] Created: (HBASE-3614) Expose per-region request rate metrics
Expose per-region request rate metrics
--------------------------------------
Key: HBASE-3614
URL: https://issues.apache.org/jira/browse/HBASE-3614
Project: HBase
Issue Type: Improvement
Components: metrics, regionserver
Reporter: Gary Helmling
Priority: Minor
We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258011#comment-13258011 ]
stack commented on HBASE-3614:
------------------------------
@Todd This issue just exposes metrics that were already being collected per region. I believe its over the metrics reporting period (5 seconds?). Want that changed? Metrics could do w/ a revamp/edit for sure.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257847#comment-13257847 ]
Hadoop QA commented on HBASE-3614:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523401/HBASE-3614-8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1582//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1582//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1582//console
This message is automatically generated.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3614:
-------------------------
Status: Patch Available (was: Open)
Trying against hadoopqa
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258394#comment-13258394 ]
Elliott Clark commented on HBASE-3614:
--------------------------------------
@Todd the time period is configurable. It's set through the hadoop-metrics hbase.period. Whenever the thread that publishes metrics comes around the averages are re-set.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-3.patch
More tests. Still needs the confs.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-7.patch
Rename the field in HRegion to mirror the new class name.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-8.patch
Fixed the spelling.
The null is there because we can get a multi put where each of the puts is for different cf's. Since we only time the total operation we're not sure which cf to assign the operation time to. It was put in __unknown previously.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257797#comment-13257797 ]
Elliott Clark commented on HBASE-3614:
--------------------------------------
And I beat you to the rename :-) is opMetrics alright.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-6.patch
Added the final to OperationMetrics.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257927#comment-13257927 ]
Hudson commented on HBASE-3614:
-------------------------------
Integrated in HBase-TRUNK #2788 (See [https://builds.apache.org/job/HBase-TRUNK/2788/])
HBASE-3614 Expose per-region request rate metrics (Revision 1328140)
Result = FAILURE
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/main/resources/hbase-default.xml
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256149#comment-13256149 ]
Elliott Clark commented on HBASE-3614:
--------------------------------------
I think per region and per cf metrics being configurable is probably the right way to go. I'll add that in the next go around.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255304#comment-13255304 ]
stack commented on HBASE-3614:
------------------------------
FYI 100 chars per line max and space around operators (this won't fly: "cfSetConsistent?cfSet:null")
I like how you are removing metrics stuff from HRegion out to a region scoped metrics class.
'+public class RegionMetrics {' needs a class comment saying what its all about. Does the class need to be public? Can it be scoped to this package only?
Collect all the data members at the top of the class. Thats whats usually done in this code base.
So put the tablename etc. in RegionMetric before the constructor etc. rather than after.
Does this need to be public generateRegionMetricsPrefix?
What do these new metrics look like? Is this all it takes to expose them?
Some regionnames are going to be really long. Should you use the region encoded name instead of the full name? Do you think we even need the table name as prefix?
Good stuff Elliott.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-0.patch
This is the first cut at getting per region metrics into the metrics framework. It adds metrics related to get/put/multiput/delete/icv. It does not add metrics around fsread times yet.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark reassigned HBASE-3614:
------------------------------------
Assignee: Elliott Clark
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261219#comment-13261219 ]
Hudson commented on HBASE-3614:
-------------------------------
Integrated in HBase-0.94-security #20 (See [https://builds.apache.org/job/HBase-0.94-security/20/])
HBASE-5836 Backport per region metrics from HBASE-3614 to 0.94.1 (Revision 1329958)
Result = SUCCESS
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionMetricsStorage.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257226#comment-13257226 ]
stack commented on HBASE-3614:
------------------------------
This could be final: '+ private RegionOperationMetrics regionMetrics;'?
100 chars per line.
Just pass HRegionInfo altogether to the below?
{code}
+ this.regionMetrics = new RegionOperationMetrics(conf, this.regionInfo.getTableNameAsString(), this.regionInfo.getEncodedName());
{code}
Err... your replacement is better than what was there previously in the below:
{code}
- final String metricPrefix = SchemaMetrics.generateSchemaMetricsPrefix(
- getTableDesc().getNameAsString(), familyMap.keySet());
- if (!metricPrefix.isEmpty()) {
- RegionMetricsStorage.incrTimeVaryingMetric(metricPrefix + "delete_", after - now);
- }
+ this.regionMetrics.updateDeleteMetrics(familyMap.keySet(), after-now);
{code}
Whats happening here?
{code}
+ if (cfSet == null) {
+ cfSet = put.getFamilyMap().keySet();
+ } else {
+ cfSetConsistent = cfSetConsistent && put.equals(cfSet);
{code}
Do we have to get the column family set each time through? It never changes (currently) while the region is open.
Whats a cfSetConsistent? A comment would help?
Yeah, I don't follow this stuff:
{code}
+ //See if the column families were consistent through the whole thing.
+ //if they were then keep them. If they were not then pass a null.
+ //null will be treated as unknown.
{code}
Should be hbase.metrics.region.exposeOperationTimes instead of hbase.metrics.exposeOperationTimes to convey its on/off for per-region metrics?
This patch is great.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258013#comment-13258013 ]
Todd Lipcon commented on HBASE-3614:
------------------------------------
Gotcha. Apologies for my cluelessness :) Seems fine. Though I think Shaneal's percentile stuff would be a nice improvement, what we've got now is at least useful.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Enis Soztutar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256122#comment-13256122 ]
Enis Soztutar commented on HBASE-3614:
--------------------------------------
This is fine for JMX, but these per-region metrics will be exposed via the Hadoop metrics system as well right? If so, the large number of regions and the fact that regions are dynamic would make it useless for ganglia (or similar). So, I think we should either not expose them through the hadoop metrics, but do itJMX only, or make per-region metrics configurable IMO.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261015#comment-13261015 ]
Hudson commented on HBASE-3614:
-------------------------------
Integrated in HBase-0.94 #145 (See [https://builds.apache.org/job/HBase-0.94/145/])
HBASE-5836 Backport per region metrics from HBASE-3614 to 0.94.1 (Revision 1329958)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionMetricsStorage.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257978#comment-13257978 ]
Todd Lipcon commented on HBASE-3614:
------------------------------------
Just getting to this after it's committed.. but: what's the time scale of "avgtime" on these metrics? Averages that are since boot are kind of useless. We should either track the sum of the time, or do a time-biased metric like Shaneal added recently in other places.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: Screen Shot 2012-04-17 at 2.41.27 PM.png
Screen shot of jmx stats after some of Stack's comments.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-4.patch
Here's a better version. It has the ability to remove the metrics.
get, set, increment, append, and delete all report their metrics per cf and per region with a pretty good api.
@Enis you're absolutely correct about SchemaMetrics needing to get static state out. I'll file an issue and try and get some time to take a look.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-2.patch
More progress.
This is not yet ready for integration. It needs tests to cover the new metrics.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258022#comment-13258022 ]
Hudson commented on HBASE-3614:
-------------------------------
Integrated in HBase-TRUNK-security #176 (See [https://builds.apache.org/job/HBase-TRUNK-security/176/])
HBASE-3614 Expose per-region request rate metrics (Revision 1328140)
Result = FAILURE
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/OperationMetrics.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
* /hbase/trunk/src/main/resources/hbase-default.xml
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-5.patch
Added more comments about multi-put.
Fixed a bug that Stack was asking about. (left off getFamilyMap().keySet() when comparing cf sets)
Renamed RegionOperationMetrics to OperationMetrics since it also exposes metrics per cf.
Added more tests around Multiputs
I didn't rename the conf key since it turns off all reporting of operation timing metrics and not just those for cf's.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3614) Expose per-region request rate
metrics
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004817#comment-13004817 ]
Jonathan Gray commented on HBASE-3614:
--------------------------------------
I'm not sure if there is a JIRA yet, but some guys at FB did a bunch of work on doing per-family metrics. They did work to dynamically generate new metric names, etc.
I think we could work on this at the same time we start to think about using the info for better load balancing and such. This could obviously come first.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Priority: Minor
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3614:
-------------------------
Resolution: Fixed
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk (936 tests passed, about same as in previous hangs). Thanks for the patch Elliott. Let me open an issue against 0.94.1 for backporting to see if we can get it in there.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-9.patch
Address the findbugs warning in HRegion. Not really a concern as that constructor is only used in unit tests.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate
metrics
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257787#comment-13257787 ]
stack commented on HBASE-3614:
------------------------------
Since you renamed RegionOperationMetrics, is this right now:
{code}
+ private final OperationMetrics regionMetrics;
{code}
Should it be named metrics or operationMetrics?
Whats 'unknown' in the following? + //null will be treated as unknown.
We are updating metrics w/o attributing them to a cf?
Fix misspell 'Inctement' in hbase-site change
Patch is good to go after addressing above. Good stuff.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, Screen Shot 2012-04-17 at 2.41.27 PM.png
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3614) Expose per-region request rate
metrics
Posted by "Elliott Clark (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-3614:
---------------------------------
Attachment: HBASE-3614-1.patch
This patch fixes the sizing tests. HRegion gets one new object reference.
> Expose per-region request rate metrics
> --------------------------------------
>
> Key: HBASE-3614
> URL: https://issues.apache.org/jira/browse/HBASE-3614
> Project: HBase
> Issue Type: Improvement
> Components: metrics, regionserver
> Reporter: Gary Helmling
> Assignee: Elliott Clark
> Priority: Minor
> Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch
>
>
> We currently export metrics on request rates for each region server, and this can help with identifying uneven load at a high level. But once you see a given server under high load, you're forced to extrapolate based on your application patterns and the data it's serving what the likely culprit is. This can and should be much easier if we just exported request rate metrics per-region on each server.
> Dynamically updating the metrics keys based on assigned regions may pose some minor challenges, but this seems a very valuable diagnostic tool to have available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira