You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Phabricator (Updated) (JIRA)" <ji...@apache.org> on 2012/02/16 02:49:01 UTC
[jira] [Updated] (HBASE-5347) GC free memory management in Level-1
Block Cache
[ https://issues.apache.org/jira/browse/HBASE-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5347:
-------------------------------
Attachment: D1635.5.patch
khemani updated the revision "[HBASE-5347] [jira] GC free memory management in Level-1 Block Cache".
Reviewers: Kannan, mbautin, dhruba, nspiegelberg, stack, JIRA
In this iteration I have been testing these changes by starting in the region server a 1000 threads that continuously call incrementColumnValue() on large random keys. In this patch there are some temporary changes to HRegionInterface to do this testing ...
In the test that I run on my dev cluster, there is enough data in the test table, the java heap is set to 24GB, block-cache is set to 60% of the heap. Without this patch I pretty soon run into GC pauses. With this patch, and even without any blocks-out-of-large-slab allocation and without changing to synchronous eviction in lru block cache, there are no GC pauses.
So, there are two kinds of objects being reference counted in this patch - HFileBlocks and KeyValues. HFileBlocks are easy to reference counts because they only exist in HFileScanners and below. KeyValues (those ones who directly refer to data in HFileBlock) are more difficult to reference count.
My earlier approach was that the StoreFileScanner will hand out pinned key-values and the higher layers will not have to bother about deref()ing the key-values. The key-values will be deref()'d only when they are written out. But this approach has many issues and quickly becomes non-intuitive. For example, with that older approach, StoreFileScanner will not be able to deref() its cur kv when it is closed and it will have to rely on the GC for cleanup.
In the current approach you don't have to ref() the kv that you get from a scanner if you are going to keep it beyond the next scanner's next(), seek(), reseek() or close() call. (If you don't ref it then you will soon get a nullptrexception). You should also deref it when you are done, but it is not absolutely required.
I have lot more cleanup to do. I will revert some of the method name changes. I will refactor the code in new classes. There is lot many tests to be written. I will re-write this in a way such that it can be run with or without the hfile-block-pool. The block pool enhancements and lru-block cache enhancements in separate diffs.
This changes the programming model quite a bit ...
Some number from the test that I am running
totKVRef 49046639
totKVDeref 49022691
totKVDead 17
totKVDeadDeadDead 1839873
49 million KVs have been reference counted. Out of that somehow the code forgets to deref 17 of them. About 2 million of them end up on the deadKVs ReferenceQueue even though they have been properly deref'd ... don't know why this happens.
curHFBPoolBlocks 165746
curHFBPoolCapacity 11541225472
curHFBPoolFreeBlocks 22050
curHFBPoolFreeCapacity 1535385600
The block cache has 10 GB of active blocks. 1.5GB is maintained in the free pool by the eviction process. We will be able to use all the block cache memory (that is around 14.5GB) once LRUBlockCache eviction is made synchronous.
totHFBPoolSystemAlloc 235873
totHFBPoolSystemFree 60075
totHFBPoolBlocksAllocated 165746
totHFBPoolLargeBlocksAllocated 70127
totHFBPoolLargeBlocksFreed 60075
totHFBPoolReuse 5459025
235K allocations were done from the system. Some of these were blocks larger than 69K so they were not managed by the pool. The blocks got resused 5.5m times.
216732
REVISION DETAIL
https://reviews.facebook.net/D1635
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/KeyValue.java
src/main/java/org/apache/hadoop/hbase/client/HTable.java
src/main/java/org/apache/hadoop/hbase/client/Result.java
src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileScanner.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
src/main/java/org/apache/hadoop/hbase/util/Counters.java
src/main/java/org/apache/hadoop/hbase/util/DoublyLinkedListElement.java
src/main/java/org/apache/hadoop/hbase/util/DoublyLinkedListHead.java
src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> GC free memory management in Level-1 Block Cache
> ------------------------------------------------
>
> Key: HBASE-5347
> URL: https://issues.apache.org/jira/browse/HBASE-5347
> Project: HBase
> Issue Type: Improvement
> Reporter: Prakash Khemani
> Assignee: Prakash Khemani
> Attachments: D1635.5.patch
>
>
> On eviction of a block from the block-cache, instead of waiting for the garbage collecter to reuse its memory, reuse the block right away.
> This will require us to keep reference counts on the HFile blocks. Once we have the reference counts in place we can do our own simple blocks-out-of-slab allocation for the block-cache.
> This will help us with
> * reducing gc pressure, especially in the old generation
> * making it possible to have non-java-heap memory backing the HFile blocks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira