[jira] [Commented] (HBASE-15561) See how G1GC works with MSLAB and chunk pool

["ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544437#comment-15544437 ] 

ramkrishna.s.vasudevan commented on HBASE-15561:
------------------------------------------------

Just wanted to update this JIRa with our findings and result over here. If things are discussed we could close this JIRA.

There are two parts to this
-> testing trunk (default memstore) with MSLAB and without MSLAB with G1GC configs
-> Next is to test G1GC with defaut memstore and offheap memstore (HBASE-15179). For now this offheap is not in trunk. It is a POC work. Working on getting things committed to trunk.

cluster setup
==========
The set up is a single node set up with one RS and the client in the same machine. The machine has 150G of RAM.

GC config:
========
export HBASE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1HeapWastePercent=20  -XX:G1MixedGCCountTarget=8"

<property>
            <name>hbase.regionserver.global.memstore.size</name>
  <value>0.42</value>
</property>


PE command:
===========
./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 --size=150 --columns=50 --valueSize=200  --oneCon=true  --writeToWAL=false --bloomFilter=NONE --inmemoryCompaction=false randomWrite 100

We think MSLAB and chunk pool is important mainly because when you move things to offheap we need to ensure that we are managing the offheap buffers 
efficiently. It is very dangerous to allocate offheap buffers and keep them unclaimed which will lead to offheap memory leak. Whereas in onheap buffers cases we
are able to manage it with the help of GC.
Hence it is always recommended for offheap memstore to go with MSLAB and chunk pool.

Perf readings
==========
For details on onheap and offheap memstore comparison with MSLAB and chunk pool pls refer to these docs and various GC configs
https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMuMaJqaMNRI86C4k5S1rQ/edit
https://docs.google.com/document/d/1bYIkwNVVyYk8bgik-C5x_yc_2hWHyi2-9KMwEFWLfzY/edit

The above experiments were done on top of a POc branch where we had our own changes to support offheap memstore including some PB changes.


With the current trunk we performed experiments with 50 threads and 100 threads and with 64 G heap memory.

With 100 threads and 10 cols per row pushed from a single client the setup with MSLAB and chunk pool performs 13% better than the no MSLAB case.
With 50 threads and 10 cols per row pushed from a single client the setup with MSLAB and chunk pool performs 11% better than the no MSLAB case.

PE completion time for 50 threads, 10 cols
||Default memstore with MSLAB ||Default memstore with NO MSLAB||
|1132.923 secs|1262 secs|

PE completion time for 100 threads, 10 cols
||Default memstore with MSLAB ||Default memstore with NO MSLAB||
|1947.330 secs|2214.201 secs|

No of GC pauses for 100 threads
||Default memstore with MSLAB ||Default memstore with NO MSLAB||
|403|475|

GC pause in secs for 100 threads
||Default memstore with MSLAB ||Default memstore with NO MSLAB||
|181.58 secs|239.9 secs|

Pls feel free to share your feedback or thoughts on this. It helps to see if there is something to be tested or some configs needs to be changed and the tests has to be repeated. 

> See how G1GC works with MSLAB and chunk pool
> --------------------------------------------
>
>                 Key: HBASE-15561
>                 URL: https://issues.apache.org/jira/browse/HBASE-15561
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>
> Based on the discussion in https://issues.apache.org/jira/browse/HBASE-14613, we need to identify specifically how MSLAB and G1GC work. This sub-task is mainly to identify how things work with G1GC and note down the observations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)