[jira] [Commented] (HBASE-15619) Performance regression observed: Random read(get) performance of branch-1 worse than 0.98

["Yu Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232622#comment-15232622 ] 

Yu Li commented on HBASE-15619:
-------------------------------

We're still trying hard to locate the problem, and will share our findings here.

There're quite some changes from 0.98.12 to 1.2.1, so I'm expecting some help from community. Any hint on what change might cause the regression or any suggestion would be highly appreciated. Thanks!

> Performance regression observed: Random read(get) performance of branch-1 worse than 0.98
> -----------------------------------------------------------------------------------------
>
>                 Key: HBASE-15619
>                 URL: https://issues.apache.org/jira/browse/HBASE-15619
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Yu Li
>            Assignee: Yu Li
>
> As titled, I observed the perf regression in the final stress testing before upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>      1) 1.1.2 with important perf fixes/improvements including HBASE-15031 and HBASE-14465 (we planed to upgrade to 1.x since Oct. last year, by when 1.1.2 was the latest stable release in branch-1)
>      2) 1.1.4 release
>      3) 1.2.1RC1
> 2. Test environment
>     * YCSB: 0.7.0 with [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
>     * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 100 threads
>     * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
>     * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same hardware for client and server
> 3. Test cases
>     * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
>     * case #1: read against empty table
>     * case #2: lrucache 100% hit
>     * case #3: BLOCKCACHE=>false
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|383562|257493|47594|
> |1.1.2+|363050|232757|35872|
> * AverageLatency(us)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|2774|4134|22371|
> |1.1.2+|2930|4572|29690|
> It seems to me each part on the read path has perf regression: RPCServer for case#1, lrucache read for case#2, and hfile read for case#3...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)