[jira] [Commented] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring

["Yu Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034265#comment-16034265 ] 

Yu Li commented on HBASE-15160:
-------------------------------

Here is the performance testing result:
|| Case || Round || Throughput (ops/s)|| AverageLatency(us)||
| w/o patch| 1 | 120820.29 | 26309.65 |
| | 2|122079.26|26019.93|
|w/ patch v6| 1 | 85544.53| 37222.54|
| |2|87071.61|36563.49|


Test details:
{noformat}
# HBase Settings
-Xmx49152m -Xms49152m -Xmn6144m -XX:SurvivorRatio=2
hfile.block.cache.size => 0.16
hbase.regionserver.handler.count => 192
hbase.rpc.server.impl => org.apache.hadoop.hbase.ipc.NettyRpcServer

# YCSB settings
recordcount=11000000 (11M)
fieldcount=1
fieldlength=1024
table schema：{NAME => 'cf', DATA_BLOCK_ENCODING => 'DIFF', VERSIONS=> '1', COMPRESSION => 'SNAPPY', IN_MEMORY => 'false', BLOCKCACHE => 'true'},{SPLITS => (1..9).map {|i| "user#{1000+i*(9999-1000)/9}"}, METADATA => {'hbase.hstore.block.storage.policy' => 'ALL_SSD'}}

LRUCache hit ratio: 93%
{noformat}

Checking the patch in more details, I doubt the regression comes from the {{System.nanoTime}} call, and will change it to {{System.currentTimeMillis}} and re-test

> Put back HFile's HDFS op latency sampling code and add metrics for monitoring
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-15160
>                 URL: https://issues.apache.org/jira/browse/HBASE-15160
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 2.0.0, 1.1.2
>            Reporter: Yu Li
>            Assignee: Yu Li
>            Priority: Critical
>         Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, hbase-15160_v6.patch
>
>
> In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, fsPreadLatency and fsWriteLatency, have been removed. There was some discussion about putting them back in a new JIRA but never happened. According to our experience, these metrics are useful to judge whether issue lies on HDFS when slow request occurs, so we propose to put them back in this JIRA, and add the metrics for monitoring as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)