What does readRequestCount metric include in addition to get and scanNext operations?

[Hakan Baba <ah...@gmail.com>]

In hbase region and regionserver metrics I cannot understand the
relationships between metrics related to read requests, get and scanNext
operations.

The hbase.regionserver.readRequestCount is documented here
<https://hbase.apache.org/book.html#rs_metrics> as:

hbase.regionserver.readRequestCount

The number of read requests received

In addition to that, there are also get_num_ops and scanNext_num_ops metrics.
I cannot find an official documentation for them. They are mentioned in
this ticket <https://issues.apache.org/jira/browse/HBASE-15376>.

In the metrics I see, the readRequests are orders of magnitude larger than
the sum of all get and scanNext requests in one regionserver. For
example:[image:
ReadGetScanMetrics] <https://i.stack.imgur.com/3fyGk.png>

Please note that the y-axis is in log scale.

The plots are using these Wavefront queries

ReadRequestRate

sum(rate(ts(hbase.regionserver.server.readRequestCount,
source="*<sourceSelector>*" )))

GetNumRate

sum(rate(ts(hbase.regionserver.regions.get_num_ops,
source="*<sourceSelector>*")))

ScanNumRate

sum(rate(ts(hbase.regionserver.regions.scanNext_num_ops,
source="*<sourceSelector>*" )))

What may be the reason of that huge delta ? What does readRequest counter
include in addition to get and scanNext operations ?

Some unproven theories of mine.

   1. Maybe the readRequests counter increments for each row read during a
   scanNext call ?
   2. Maybe the readRequest counter increments with async replication jobs,
   region replication, cross datacenter replication etc.

Note this is running HBase 1.2.0-cdh5.11.0


You can see this question also in server fault, if you like to see a
different formatting.

https://serverfault.com/q/973638/428655

Thanks

Hakan

Re: What does readRequestCount metric include in addition to get and scanNext operations?

[Ankit Singhal <an...@gmail.com>]

>
> Maybe the readRequests counter increments for each row read during a
> scanNext call ?

Correct!!
https://github.com/apache/hbase/blob/rel/1.2.12/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L2672
https://github.com/apache/hbase/blob/rel/1.2.12/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1134

Maybe the readRequest counter increments with async replication jobs,
> region replication, cross datacenter replication etc.

readRequest should not be impacted due to replication.

Regards,
Ankit Singhal

On Thu, Jul 4, 2019 at 6:59 PM Hakan Baba <ah...@gmail.com> wrote:

> In hbase region and regionserver metrics I cannot understand the
> relationships between metrics related to read requests, get and scanNext
> operations.
>
> The hbase.regionserver.readRequestCount is documented here
> <https://hbase.apache.org/book.html#rs_metrics> as:
>
> hbase.regionserver.readRequestCount
>
> The number of read requests received
>
> In addition to that, there are also get_num_ops and scanNext_num_ops
> metrics.
> I cannot find an official documentation for them. They are mentioned in
> this ticket <https://issues.apache.org/jira/browse/HBASE-15376>.
>
> In the metrics I see, the readRequests are orders of magnitude larger than
> the sum of all get and scanNext requests in one regionserver. For
> example:[image:
> ReadGetScanMetrics] <https://i.stack.imgur.com/3fyGk.png>
>
> Please note that the y-axis is in log scale.
>
> The plots are using these Wavefront queries
>
> ReadRequestRate
>
> sum(rate(ts(hbase.regionserver.server.readRequestCount,
> source="*<sourceSelector>*" )))
>
> GetNumRate
>
> sum(rate(ts(hbase.regionserver.regions.get_num_ops,
> source="*<sourceSelector>*")))
>
> ScanNumRate
>
> sum(rate(ts(hbase.regionserver.regions.scanNext_num_ops,
> source="*<sourceSelector>*" )))
>
> What may be the reason of that huge delta ? What does readRequest counter
> include in addition to get and scanNext operations ?
>
> Some unproven theories of mine.
>
>    1. Maybe the readRequests counter increments for each row read during a
>    scanNext call ?
>    2. Maybe the readRequest counter increments with async replication jobs,
>    region replication, cross datacenter replication etc.
>
> Note this is running HBase 1.2.0-cdh5.11.0
>
>
> You can see this question also in server fault, if you like to see a
> different formatting.
>
> https://serverfault.com/q/973638/428655
>
> Thanks
>
> Hakan
>