You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/07/16 02:03:50 UTC

[jira] Created: (HBASE-2838) Replication metrics

Replication metrics
-------------------

                 Key: HBASE-2838
                 URL: https://issues.apache.org/jira/browse/HBASE-2838
             Project: HBase
          Issue Type: Sub-task
            Reporter: Jean-Daniel Cryans
            Assignee: Jean-Daniel Cryans




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892558#action_12892558 ] 

HBase Review Board commented on HBASE-2838:
-------------------------------------------

Message from: "Jean-Daniel Cryans" <jd...@apache.org>


bq.  On 2010-07-24 21:01:02, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java, line 142
bq.  > <http://review.hbase.org/r/381/diff/1/?file=3223#file3223line142>
bq.  >
bq.  >     There will always be edits if we get this far?  (I'm wondering if ever a illegalAccessException possible here)
bq.  >

I'll put a safeguard at the beginning of the method to ensure we have a least 1 entry. It should always be the case... but you never


bq.  On 2010-07-24 21:01:02, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java, line 504
bq.  > <http://review.hbase.org/r/381/diff/1/?file=3224#file3224line504>
bq.  >
bq.  >     Should this be 'notted'?  You want to count when row keys are not the same?

/me facedesk


bq.  On 2010-07-24 21:01:02, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationStatistics.java, line 32
bq.  > <http://review.hbase.org/r/381/diff/1/?file=3226#file3226line32>
bq.  >
bq.  >     Do you have to do this to make them show in JMX?  I didn't know.

No idea, but the rest of the metrics stuff is done like that...


bq.  On 2010-07-24 21:01:02, stack wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java, line 183
bq.  > <http://review.hbase.org/r/381/diff/1/?file=3227#file3227line183>
bq.  >
bq.  >     What this?  Reenabling tests?

they were disabled in another commit inadvertently, re-enabling.


bq.  On 2010-07-24 21:01:02, stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationMetrics.java, line 42
bq.  > <http://review.hbase.org/r/381/diff/1/?file=3222#file3222line42>
bq.  >
bq.  >     For sure these metrics show fine in ganglia?  Whats the prefix that they show up under?  Is there one? (Looks like its hbase?  Thats OK i'd say.  Should prefix have replication in it somewhere though?)

It is prefixed with hbase.replication automatically.


- Jean-Daniel


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/381/#review481
-----------------------------------------------------------





> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892565#action_12892565 ] 

HBase Review Board commented on HBASE-2838:
-------------------------------------------

Message from: "Jean-Daniel Cryans" <jd...@apache.org>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/381/
-----------------------------------------------------------

(Updated 2010-07-26 17:24:25.230486)


Review request for hbase.


Changes
-------

I split the replication metrics into two classes (one for source, one for sink). ReplicationSourceMetrics will be instantiated as many times as there are sources, and they are named with the source id. Took care of the other comments.


Summary
-------

Added the basic metrics, currently only accessible via hadoop metrics. Metrics maintenance is done inline using ReplicationMetrics as a Singleton, wanted to get others opinion on that.

If we are to cut a 0.89 soon, at least this patch needs to go in IMO. 


This addresses bug HBASE-2838.
    http://issues.apache.org/jira/browse/HBASE-2838


Diffs (updated)
-----

  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java 3eec1d7 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkMetrics.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 855cd6f 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java 0501048 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationStatistics.java PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 28a6e51 

Diff: http://review.hbase.org/r/381/diff


Testing
-------

Unit tests


Thanks,

Jean-Daniel




> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892624#action_12892624 ] 

HBase Review Board commented on HBASE-2838:
-------------------------------------------

Message from: stack@duboce.net

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/381/#review495
-----------------------------------------------------------

Ship it!


+1 on commit (if tests pass)

- stack





> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2838) Replication metrics

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans updated HBASE-2838:
--------------------------------------

    Fix Version/s: 0.90.0
      Description: 
Replication needs to publish metrics about its performance:

 - WALEdits read, filtered, sent to slave clusters, applied on slaves
 - size of batches sent/received
 - ms spent on reading, sending, applying edits

This can be done using HadoopMetrics.

Also we need to publish information not related to performance:

 - size of each HLog queues
 - age of the last replicated edit in each queue
 - time of last successful replication

These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892070#action_12892070 ] 

HBase Review Board commented on HBASE-2838:
-------------------------------------------

Message from: stack@duboce.net

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/381/#review481
-----------------------------------------------------------


Looks good.  Few questions in below.


src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationMetrics.java
<http://review.hbase.org/r/381/#comment1975>

    For sure these metrics show fine in ganglia?  Whats the prefix that they show up under?  Is there one? (Looks like its hbase?  Thats OK i'd say.  Should prefix have replication in it somewhere though?)



src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
<http://review.hbase.org/r/381/#comment1976>

    There will always be edits if we get this far?  (I'm wondering if ever a illegalAccessException possible here)
    



src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
<http://review.hbase.org/r/381/#comment1977>

    Should this be 'notted'?  You want to count when row keys are not the same?



src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationStatistics.java
<http://review.hbase.org/r/381/#comment1978>

    Do you have to do this to make them show in JMX?  I didn't know.



src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
<http://review.hbase.org/r/381/#comment1979>

    What this?  Reenabling tests?


- stack





> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-2838) Replication metrics

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans resolved HBASE-2838.
---------------------------------------

    Hadoop Flags: [Reviewed]
      Resolution: Fixed

Committed to trunk, thanks for reviewing Stack!

> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888980#action_12888980 ] 

ryan rawson commented on HBASE-2838:
------------------------------------

you can publish the other stats via hadoop metrics as well.  dont
publish the long of how old the longest one is, but publish the delay
time, ie: the time difference. in a graph normally this will hover
near 0, but during times of trouble it may climb thus making a clear
indication that something is wrong.

Another metric you can track is queue linger time - how long do items
remain in various queues before being processed.  You'd probably have
to track and average this.

On Thu, Jul 15, 2010 at 5:42 PM, Jean-Daniel Cryans (JIRA)



> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889440#action_12889440 ] 

Gary Helmling commented on HBASE-2838:
--------------------------------------

I'm guessing these would be added in as additional attributes in o.a.h.h.regionserver.metrics.RegionServerMetrics?

For the metrics values, look at:

o.a.h.h.metrics.MetricsRate
 - averaged rate of occurrances per time unit (requests/sec)

o.a.hadoop.metrics.util.MetricsTimeVaryingInt
 - counter reset to 0 at end of sample period

o.a.hadoop.metrics.util.MetricsTimeVaryingLong
 - same as a long counter

o.a.hadoop.metrics.util.MetricsTimeVaryingRate
 - time-based rate reset to 0 at end of sample period (RPC queue time)

These should map pretty well to the values you want to publish.  The time-varying ones also expose min and max values for the sample period (or since last reset call) as well.


> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2838) Replication metrics

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891869#action_12891869 ] 

HBase Review Board commented on HBASE-2838:
-------------------------------------------

Message from: "Jean-Daniel Cryans" <jd...@apache.org>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/381/
-----------------------------------------------------------

Review request for hbase.


Summary
-------

Added the basic metrics, currently only accessible via hadoop metrics. Metrics maintenance is done inline using ReplicationMetrics as a Singleton, wanted to get others opinion on that.

If we are to cut a 0.89 soon, at least this patch needs to go in IMO. 


This addresses bug HBASE-2838.
    http://issues.apache.org/jira/browse/HBASE-2838


Diffs
-----

  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationMetrics.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java 3eec1d7 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 855cd6f 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java 0501048 
  src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationStatistics.java PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 28a6e51 

Diff: http://review.hbase.org/r/381/diff


Testing
-------

Unit tests


Thanks,

Jean-Daniel




> Replication metrics
> -------------------
>
>                 Key: HBASE-2838
>                 URL: https://issues.apache.org/jira/browse/HBASE-2838
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.90.0
>
>
> Replication needs to publish metrics about its performance:
>  - WALEdits read, filtered, sent to slave clusters, applied on slaves
>  - size of batches sent/received
>  - ms spent on reading, sending, applying edits
> This can be done using HadoopMetrics.
> Also we need to publish information not related to performance:
>  - size of each HLog queues
>  - age of the last replicated edit in each queue
>  - time of last successful replication
> These informations can hardly be graphed, but we still need to represent them somehow. It has to be accessible by web UI, shell, and other tools in general. I don't feel strongly about creating a new public method on HRS's interface, and I'm not sure publishing those in ZooKeeper is a good idea either (why add another indirection?). Still wondering about a better solution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.