You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org> on 2016/07/22 17:33:21 UTC
[jira] [Comment Edited] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389043#comment-15389043 ]
ramkrishna.s.vasudevan edited comment on HBASE-14921 at 7/22/16 5:32 PM:
-------------------------------------------------------------------------
Did the testing with compacting memstore.
Based on our write path offheaping work we had arrived at certain config params and gc tuning.
Onheap case : Xmx : 32 G
Offheap case Xmx : 12 G
Offheap memory for memstore : 25 G
global memstore size : 0.42
GC - G1GC
"-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1HeapWastePercent=20 -XX:G1MixedGCCountTarget=8"
PE : ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 --size=150 --columns=50 --valueSize=200 --writeToWAL=false --bloomFilter=NONE --inmemoryCompaction=false/true randomWrite 50
We were able to run and get results for the default memstore (onheap), default memstore(offheap), Pipelinememstore (Internal impl where all segments are flushed with no in memory compaction) - flatten to ARRAY_MAP and flatten to CHUNK_MAP.
But with compacting memstore we are not able to run and get results as with the offheap case 12G is just not enough because we have duplicates while doing in memory compaction. The segments are read once again to create a compacted segment and it is creating lot of garbage and more memory need.
With onheap case I increased the heap size to 50G (as with 32G I was not able to run) still the compaction was taking lot of memory and most importantly the disk flushes and disk compactions were getting piled up and finally got 'Too many open files' exception.
So I copied the latest trunk patch and gave a run and it had other exception due to invalid region size settings as it was going negative.
{code}
2016-07-22 16:14:51,066 ERROR [RpcServer.FifoWFPBQ.default.handler=60,queue=0,port=16041-inmemoryCompactions-1469184056544] regionserver.HRegion: Asked to modify this region's (TestTable,00000000000000000005033136,1469184040624.26d47b5614da8885fd0768b66a5d6457.) memstoreSize to a negative value which is incorrect. Current memstoreSize=33534312, delta=-33570288
java.lang.Exception
at org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1125)
at org.apache.hadoop.hbase.regionserver.RegionServicesForStores.addAndGetGlobalMemstoreSize(RegionServicesForStores.java:72)
at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:111)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:208)
at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:173)
at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:114)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:311)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
So in order to avoid compaction I just ensure that only flattening happens (because in the default case that is what you are trying to do). Even here the problem is that since only the flush happens on the tail part we keep pressurizing the memstore and we have lot of blocked regions. Also the tail flush is creating lot of small files and this is leading to more compactions and hence again getting 'Too many file open' exception.
When i did the above change to the offheap memstore (allow only flattening) case the problem is similar but here the tests are taking longer time to complete as every time only a small portion of the memstore is getting flushed out.
So we feel that if we go only with flattening then we should ensure that entire segments in pipeline has to be flushed.
was (Author: ram_krish):
Did the testing with compacting memstore.
Based on our write path offheaping work we had arrived at certain config params and gc tuning.
Onheap case : Xmx : 32 G
Offheap case Xmx : 12 G
Offheap memory for memstore : 32 G
global memstore size : 0.42
GC - G1GC
"-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=60 -XX:G1HeapWastePercent=20 -XX:G1MixedGCCountTarget=8"
PE : ./hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=50 --size=150 --columns=50 --valueSize=200 --writeToWAL=false --bloomFilter=NONE --inmemoryCompaction=false/true randomWrite 50
We were able to run and get results for the default memstore (onheap), default memstore(offheap), Pipelinememstore (Internal impl where all segments are flushed with no in memory compaction) - flatten to ARRAY_MAP and flatten to CHUNK_MAP.
But with compacting memstore we are not able to run and get results as with the offheap case 12G is just not enough because we have duplicates while doing in memory compaction. The segments are read once again to create a compacted segment and it is creating lot of garbage and more memory need.
With onheap case I increased the heap size to 50G (as with 32G I was not able to run) still the compaction was taking lot of memory and most importantly the disk flushes and disk compactions were getting piled up and finally got 'Too many open files' exception.
So I copied the latest trunk patch and gave a run and it had other exception due to invalid region size settings as it was going negative.
{code}
2016-07-22 16:14:51,066 ERROR [RpcServer.FifoWFPBQ.default.handler=60,queue=0,port=16041-inmemoryCompactions-1469184056544] regionserver.HRegion: Asked to modify this region's (TestTable,00000000000000000005033136,1469184040624.26d47b5614da8885fd0768b66a5d6457.) memstoreSize to a negative value which is incorrect. Current memstoreSize=33534312, delta=-33570288
java.lang.Exception
at org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1125)
at org.apache.hadoop.hbase.regionserver.RegionServicesForStores.addAndGetGlobalMemstoreSize(RegionServicesForStores.java:72)
at org.apache.hadoop.hbase.regionserver.CompactionPipeline.swap(CompactionPipeline.java:111)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.swapCompactedSegments(CompactingMemStore.java:208)
at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompaction(MemStoreCompactor.java:173)
at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.start(MemStoreCompactor.java:114)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory(CompactingMemStore.java:311)
at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run(CompactingMemStore.java:384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
So in order to avoid compaction I just ensure that only flattening happens (because in the default case that is what you are trying to do). Even here the problem is that since only the flush happens on the tail part we keep pressurizing the memstore and we have lot of blocked regions. Also the tail flush is creating lot of small files and this is leading to more compactions and hence again getting 'Too many file open' exception.
When i did the above change to the offheap memstore (allow only flattening) case the problem is similar but here the tests are taking longer time to complete as every time only a small portion of the memstore is getting flushed out.
So we feel that if we go only with flattening then we should ensure that entire segments in pipeline has to be flushed.
> Memory optimizations
> --------------------
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: Eshcar Hillel
> Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)