[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526106#comment-13526106 ] 

Sergey Shelukhin commented on HBASE-6466:
-----------------------------------------

Hmm, sorry for long absence on this issue.
I ran some tests on EC2, with 1 RS and 5 DataNode-s with one mount point each.
EC2 was probably not the best choice for testing disk, so I was only able to see perf improvement under artificial conditions.
I am guessing that would also happen if I used enough real disks...
20 regions for LoadTestTool on the RS, no compactions during the test, WAL enabled, large number of IPC handlers.
I set the flush limit to 1Mb to trigger a lot of flushes and used LoadTestTool to write ~200k values, most flushes were on the order of a few Mb.
LTT is getting about 100 writes/sec with 1 flush handler, and I can see from iostat disks on some of the datanodes are idling (different at different time obviously).
With 3 or 5 handlers disks are all busy and I get about 150 writes/sec.
                
> Enable multi-thread for memstore flush
> --------------------------------------
>
>                 Key: HBASE-6466
>                 URL: https://issues.apache.org/jira/browse/HBASE-6466
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>         Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.1.patch, HBASE-6466v3.patch
>
>
> If the KV is large or Hlog is closed with high-pressure putting, we found memstore is often above the high water mark and block the putting.
> So should we enable multi-thread for Memstore Flush?
> Some performance test data for reference,
> 1.test environment ： 
> random writting；upper memstore limit 5.6GB;lower memstore limit 4.8GB;400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 thread handler per client for writing
> 2.test results:
> one cacheFlush handler, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver, appears many aboveGlobalMemstoreLimit blocking
> two cacheFlush handlers, tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver,
> 200 thread handler per client & two cacheFlush handlers, tps:16.1k/s per regionserver, Flush:18.6MB/s per regionserver

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira