[jira] [Commented] (HBASE-23617) Add a stress test tool for region based procedure store

["Duo Zhang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-23617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003547#comment-17003547 ] 

Duo Zhang commented on HBASE-23617:
-----------------------------------

I changed the default config for RegionProcedureStore a bit, so now the default flush size and log roller size are all 128MB, and I also changed the roll size for WALProcedureStore to 128MB.

For hflush
{noformat}
./bin/hbase org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStorePerformanceEvaluation &>test_region.log < /dev/null &
./bin/hbase org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALPerformanceEvaluation &>test_wal.log < /dev/null &
{noformat}

The result is pretty good, for RegionProcedureStore
{noformat}
Num threads    : 20
Num procedures : 1000000
Sync type      : hflush
Time taken     : 52sec
{noformat}

For WALProcedureStore
{noformat}
Num threads    : 20
Num procedures : 1000000
Sync type      : hflush
Time taken     : 90sec
{noformat}

But when using hsync, the RegionProcedureStore became extremely slow, need to dig more.

For one thread test, RegionProcedureStore is still faster.
{noformat}
./bin/hbase org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStorePerformanceEvaluation -sync hsync -threads 1 -procs 100000 &>test_region_hsync.log < /dev/null &
./bin/hbase org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALPerformanceEvaluation -sync hsync -threads 1 -procs 100000 &>test_wal_hsync.log < /dev/null &
{noformat}

For RegionProcedureStore
{noformat}
Num threads    : 1
Num procedures : 100000
Sync type      : hsync
Time taken     : 73sec
{noformat}
For WALProcedureStore
{noformat}
Num threads    : 1
Num procedures : 100000
Sync type      : hsync
Time taken     : 133sec
{noformat}

But the story changes when we use 20 threads to write 1000000 procedures...

Maybe there is something wrong in our HRegion implementation when using hsync...

> Add a stress test tool for region based procedure store
> -------------------------------------------------------
>
>                 Key: HBASE-23617
>                 URL: https://issues.apache.org/jira/browse/HBASE-23617
>             Project: HBase
>          Issue Type: Improvement
>          Components: proc-v2
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>            Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)