[jira] [Commented] (HBASE-8711) Requests count is completely off

["James Kinley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777370#comment-13777370 ] 

James Kinley commented on HBASE-8711:
-------------------------------------


There are a couple of things going on here. The first issue is that the point in time metrics for a region server are only computed every 15 seconds and therefore requestsPerSecond is actually an average for the previous interval:

{code}
requestsPerSecond = (numberRequestsSinceLastRun / secondsSinceLastRun)
{code}

The client write buffer also confuses things because the writes are delayed reaching the server, then the server can take up to 15 seconds to include them in the metrics, and another 15 seconds to clear them down.

The second issue is that the requestCount counter used to compute requestsPerSecond is incremented for every region server operation (i.e. including requests for region info and not just for mutations) and is therefore higher than expected.

I've attached the results of two tests to show whats happening. The same client was used in both tests to generate 100 req/sec for 60 seconds (6000 writes):

In the first test (RPS-TEST-1.log) the client write buffer was enabled and you can see that it took 52 seconds before the writes appeared in the metrics, and because they were all sent at once requestsPerSecond was skewed at 09:07:28 (requestsPerSecond: 602.7862951606452). It took a further 30 seconds before requestsPerSecond returned to 0. Also, writeRequestsCount was only incremented once for each flush, and the individual writes were not counted (tempWriteRequestsCount = 4).

In the second test (RPS-TEST-2.log) the client write buffer was disabled and you can see that the writes started to appear in the metrics after the next metric computation was run (15 seconds), and requestsPerSecond hovered at ~170 for the duration of the test. The writes were also counted individually (tempWriteRequestsCount = 6004).

Both tests show an increase of over 12000 requests for 6000 writes.

Computing requestsPerSecond from the requestCount counter makes sense, but the interval should at least be configurable and be lower by default (maybe 5 seconds).
                
> Requests count is completely off
> --------------------------------
>
>                 Key: HBASE-8711
>                 URL: https://issues.apache.org/jira/browse/HBASE-8711
>             Project: HBase
>          Issue Type: Bug
>          Components: UI
>    Affects Versions: 0.95.1
>            Reporter: Jean-Daniel Cryans
>            Assignee: James Kinley
>             Fix For: 0.96.0
>
>
> I tried 0.95.1 RC1 in standalone, and the requests count in both the master and RS web UIs are wrong. I haven't dug too much in but it seems too low when I'm sending load, and it takes >10 seconds to clear up when the cluster becomes completely idle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira