You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2011/06/24 03:57:47 UTC
[jira] [Created] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Enable direct byte buffers LruBlockCache
----------------------------------------
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside of the heap.
They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: 4027-v5.diff
addressed all of Todd's comments. Added multi threaded tests, wrote test for Slab.java.
To Stack/Ted Yu: now is a good time to review.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072548#comment-13072548 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-07-28 21:39:13.311916)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing (updated)
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073113#comment-13073113 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1238
-----------------------------------------------------------
conf/hbase-env.sh
<https://reviews.apache.org/r/1214/#comment2812>
Still some whitespaces.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment2813>
Doesn't match actual name of class.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2814>
I don't see SkipList being used in this class.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
<https://reviews.apache.org/r/1214/#comment2815>
This doesn't match the actual test.
- Ted
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068013#comment-13068013 ]
Ted Yu commented on HBASE-4027:
-------------------------------
The code Stack mentioned:
{code}
+ contentBlock = backingStore.get(key).getBlock(key, caching);
{code}
would lead to frequent NPE.
There were only 87 entries in ConcurrentHashMap (backingStore) after cache.cacheBlock() call at line 58 in TestSlabCache
The assertion at line 64 would fail.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v12.1.diff
added heapsize, fixed formatting, addressed ted yu's reviews.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068549#comment-13068549 ]
Li Pi commented on HBASE-4027:
------------------------------
Yeah, thats what I'm working on as well. Some easy visual way to expose stats.
Time to learn webdev!
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087482#comment-13087482 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1543
-----------------------------------------------------------
Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap.
- Li
On 2011-08-19 03:05:29, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 03:05:29)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. CHANGES.txt 763ddbc
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
bq. src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072466#comment-13072466 ]
Li Pi commented on HBASE-4027:
------------------------------
gimme a sec. will do it.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v11.6.diff
fixed broken test and race condition.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: HBase4027v8.diff
Fixed build error, addressed most of Ted Yu/JGray's review.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070965#comment-13070965 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > pom.xml, line 712
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26914#file26914line712>
bq. >
bq. > did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on.
Changed to MapMaker. Guava has deprecated some of the functions I'm using in trunk, but hasn't provided a release with a replacement. But we'll deal with that when the time comes.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 33
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line33>
bq. >
bq. > punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well.
Done
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 52-55
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line52>
bq. >
bq. > No need to line-break here
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 64-65
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line64>
bq. >
bq. > consider using StringUtils.humanReadableInt for these sizes.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 72
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line72>
bq. >
bq. > @Override
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-85
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line77>
bq. >
bq. > when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /** {@inheritDoc} */ to be explicit that you're inheriting from the superclass.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 102
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line102>
bq. >
bq. > I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 128
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line128>
bq. >
bq. > hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward
Fixed. Also refactored CacheStats into a separate class.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 148-166
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line148>
bq. >
bq. > TODOs
Implemented these.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 168
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line168>
bq. >
bq. > is this code used? seems like dead copy-paste code to me.
It is now.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java, line 1125
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26916#file26916line1125>
bq. >
bq. > extraneous debugging left in
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 31
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line31>
bq. >
bq. > I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 32-34
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line32>
bq. >
bq. > unclear what the difference is between the two.
bq. >
bq. > Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items
bq. >
bq. > Also, rather than // comments, use /** javadoc comments */ before the vars
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 46
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line46>
bq. >
bq. > these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 49-55
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line49>
bq. >
bq. > I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like:
bq. >
bq. > int numFullSlabs = numBlocks / maxBlocksPerSlab;
bq. > boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0;
bq. >
bq. > for (int i = 0; i < numFullSlabs; i++) {
bq. > alloc one of maxSlabSize;
bq. > addBuffersForSlab(slab);
bq. > }
bq. >
bq. > if (hasPartialSlab) {
bq. > alloc the partial one
bq. > addBuffersForSlab(slab);
bq. > }
bq. >
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 77-78
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line77>
bq. >
bq. > should be a LOG.warn
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 43
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line43>
bq. >
bq. > shouldn't this implement BlockCache?
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 58-65
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line58>
bq. >
bq. > no need to line-break
Eclipse automatically does this. Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 67
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line67>
bq. >
bq. > It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell...
If the SingleSizeSlab decides to evict, the master should evict as well.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 70
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line70>
bq. >
bq. > I don't understand what purpose returnMap is serving here
ReturnMap is pretty much how free() is implemented. I probably should move this to MetaSlab - now Slab
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 94
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line94>
bq. >
bq. > shouldn't this class have an alloc() and free() method?
Moving alloc() and free() functions from SingleSizeCache to MetaSlab. Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 106
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line106>
bq. >
bq. > this is where you would call backingStore.alloc() -- then buffers can be made into a private member and keep encapsulation
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 123
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line123>
bq. >
bq. > @Override, remove javadoc
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 96-99
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line96>
bq. >
bq. > add '@Override', then you don't need to copy-paste javadoc
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, lines 131-133
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line131>
bq. >
bq. > this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc.
bq. >
bq. > Instead, check backingMap.get(key) against null
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 155
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line155>
bq. >
bq. > why does master need to know about this?
Master keeps its own records for which SingleSizeCache the key is assigned to.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 164
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line164>
bq. >
bq. > numBlocks isn't the size, is it?
Good catch. Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 166
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line166>
bq. >
bq. > missing space before "blocks"
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java, line 207
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26918#file26918line207>
bq. >
bq. > there are several identical copies of this floating around. Can they all use the same class?
Refactored out and fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 21
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line21>
bq. >
bq. > overall note: I think we could move all of this to a new io.hfile.slab package
Will do next revision.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 44
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line44>
bq. >
bq. > should this implement BlockCache?
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 47
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line47>
bq. >
bq. > since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient
Fixed. Although, in the future, we may want to modify it on the fly, which is why it was a ConcurrentSkipListMap.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 49
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line49>
bq. >
bq. > rename to statThreadPeriodSecs to indicate time unit.
bq. >
bq. > Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up
fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 53
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line53>
bq. >
bq. > you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below
changed to runnable.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 65-68
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line65>
bq. >
bq. > line breaks unnecessary
eclipse autoformatting :(. fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 80-85
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line80>
bq. >
bq. > this stuff will have to be configurable too. but this is fine for now
next revision!
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 95
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line95>
bq. >
bq. > should also note that it returns 0 for the case of an uncacheable object
changed so it returns the entry instead of an int. returns null if unreachable.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 98
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line98>
bq. >
bq. > this function is never used -- you should make it private, and then call it from cacheBlock below.
fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, lines 145-148
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line145>
bq. >
bq. > again this is no good. check for null instead of catching an exception
fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 174
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line174>
bq. >
bq. > when would this be true? evicting something that doesn't have an entry in backingStore?
It'll be true because theres a callback from SlabCache.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 177
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line177>
bq. >
bq. > it seems like there's a loop in flow control here.
Nope, SingleSlabCache should return false, and not loop.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 214
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line214>
bq. >
bq. > move this nice logging output to something like: slabstats.logUsage()
Done.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 229
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line229>
bq. >
bq. > since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread
Done
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 250
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line250>
bq. >
bq. > need to be public?
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 253
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line253>
bq. >
bq. > should be ALL_CAPS
Done
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 254
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line254>
bq. >
bq. > ALSO_ALL_CAPS
Done.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java, line 268
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26919#file26919line268>
bq. >
bq. > why boxed longs?
Changed to AtomicLong[]
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 369
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26920#file26920line369>
bq. >
bq. > for consistency, I think we should name these something like:
bq. > hfile.block.cache.offheap.size
bq. > hfile.block.cache.offheap.minblocksize
bq. >
bq. > also, is min block size really a min? isn't it more like expectedblocksize?
Will do next revision.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 84
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line84>
bq. >
bq. > nowhere in this test does it actually verify that the data is valid
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 38
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line38>
bq. >
bq. > should use JUnit 4 style tests in new code - i.e not inherit TestCase.
I thought since everything else was JUnit 3 style, I should leave it as JUnit 3 style.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java, line 109
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26922#file26922line109>
bq. >
bq. > see above.
bq. >
bq. > If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil
Will fix this when I redo all tests tomorrow.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java, line 95
bq. > <https://reviews.apache.org/r/1191/diff/1/?file=26921#file26921line95>
bq. >
bq. > some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache
Will fix this when I redo all tests tomorrow.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1191/#review1182
-----------------------------------------------------------
On 2011-07-25 22:55:56, Todd Lipcon wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1191/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-25 22:55:56)
bq.
bq.
bq. Review request for hbase and Todd Lipcon.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Uploading slabcachepatchv4 to review for Li Pi.
bq.
bq.
bq. This addresses bug HBASE-4027.
bq. https://issues.apache.org/jira/browse/HBASE-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. pom.xml 729dc37
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1191/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Todd
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092072#comment-13092072 ]
stack commented on HBASE-4027:
------------------------------
I get this in TRUNK at mo:
{code}
Running org.apache.hadoop.hbase.client.TestAdmin
Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 227.229 sec <<< FAILURE!
{code}
I'm taking a looksee.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072940#comment-13072940 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1211
-----------------------------------------------------------
conf/hbase-env.sh
<https://reviews.apache.org/r/1214/#comment2675>
Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment2676>
2011
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment2677>
whitespace
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment2678>
license
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment2679>
class comment
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2680>
whitespace here and throughout this file
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2681>
Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).
If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2682>
longer than 80 chars
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2683>
This seems like a behavior that we may not always want.
If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)
we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2684>
this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment2685>
huh?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment2686>
line > 80 chars
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment2687>
getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc)
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment2688>
javadoc on these
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2689>
I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2690>
Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).
Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.
I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2691>
seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2692>
is this really a cache miss? this cache will never take this block.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2693>
but here we need to check for null and increment cache miss rather than hit if it is null
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2694>
this non-javadoc comment is not necessary, just the @Override has been normal format in hbase
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment2762>
should this really be a bunch of INFO logs? Could we have these exported as metrics instead?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
<https://reviews.apache.org/r/1214/#comment2763>
license
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
<https://reviews.apache.org/r/1214/#comment2764>
formatting looks a little strange here (missing spaces and such)
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
<https://reviews.apache.org/r/1214/#comment2765>
line > 80 chars
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
<https://reviews.apache.org/r/1214/#comment2766>
2011
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
<https://reviews.apache.org/r/1214/#comment2767>
This test looks more specific than testing "the concurrent LruBlockCache"
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
<https://reviews.apache.org/r/1214/#comment2768>
you are missing a space here and throughout this file and some others... our format is to always have a space before a {
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
<https://reviews.apache.org/r/1214/#comment2769>
license
- Jonathan
On 2011-07-28 23:02:50, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-28 23:02:50)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088211#comment-13088211 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1585
-----------------------------------------------------------
In SingleSizeCache.cacheBlock():
CacheablePair newEntry = new CacheablePair(
toBeCached.serialize(storedBlock), storedBlock);
The above operation splits toBeCached into two parts: the first is for on-heap and is slim, storedBlock is for off-heap.
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3574>
I think the word 'itself' in the javadoc above introduced confusion. It should be removed.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3573>
As Pi explained in Cacheable interface, serialize() offloads majority of data to off-heap ByteBuffer. What gets returned is the skeleton that lives on-heap.
- Ted
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084428#comment-13084428 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-12 22:30:10.310736)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed two bugs as per ted yu's reviews.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: 4027v7.diff
addressed some of Ted Yu/JGrays comments on reviewboard.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075983#comment-13075983 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-02 00:46:57.567666)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
configuration changed to percentage of MaxDirectMemorySize rather than specifying size of offHeapCache.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089985#comment-13089985 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1612
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3644>
User may get confused by this javadoc since CacheableDeserializer is interface.
How about:
Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
<https://reviews.apache.org/r/1214/#comment3643>
This javadoc should be above interface declaration.
- Ted
On 2011-08-24 02:46:25, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 02:46:25)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087703#comment-13087703 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1546
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3522>
Still some white spaces.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3518>
This is an important class related to Cacheable. Should have added javadoc for class and methods.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3519>
REFERENCE refers to serializedData, right ?
I think it would be clearer if the order of second and third parameters gets exchanged.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
<https://reviews.apache.org/r/1214/#comment3521>
Can you explain the meaning of callAssignedCache ?
It is not obvious by name.
- Ted
On 2011-08-19 04:16:44, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 04:16:44)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt 0478003
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067598#comment-13067598 ]
Ted Yu commented on HBASE-4027:
-------------------------------
onHeapCache should be used in getBlock():
{code}
+ /* if ((cachedBlock = onHeapCache.getBlock(blockName, caching)) != null) {
+ return cachedBlock;
+ } else */ if ((cachedBlock = offHeapCache.getBlock(blockName, caching)) != null) {
+ // onHeapCache.cacheBlock(blockName, cachedBlock);
{code}
Otherwise onHeapCache.cacheBlock() should be commented out in:
{code}
+ public void cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) {
+ onHeapCache.cacheBlock(blockName, buf, inMemory);
+ offHeapCache.cacheBlock(blockName, buf);
+ }
{code}
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084425#comment-13084425 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-12 21:52:43, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 38
bq. > <https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38>
bq. >
bq. > Still some white spaces to remove.
Got it.
bq. On 2011-08-12 21:52:43, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37
bq. > <https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37>
bq. >
bq. > Incorrect class name.
Doh. Fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1430
-----------------------------------------------------------
On 2011-08-12 20:26:21, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 20:26:21)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072471#comment-13072471 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
Review request for hbase, Todd Lipcon and Li Pi.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v11.5.diff
fixed test failure, as reported by ted yu.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088084#comment-13088084 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-19 14:36:26, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 78
bq. > <https://reviews.apache.org/r/1214/diff/16/?file=33706#file33706line78>
bq. >
bq. > The following 10 lines of code is very similar to those starting line 65.
bq. > Should be able to write another method so that both of them can call.
fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1549
-----------------------------------------------------------
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi reassigned HBASE-4027:
----------------------------
Assignee: Li Pi
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084406#comment-13084406 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1430
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3294>
Still some white spaces to remove.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3295>
Incorrect class name.
- Ted
On 2011-08-12 20:26:21, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 20:26:21)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027-v10.diff
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091638#comment-13091638 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/25/?file=34984#file34984line1>
bq. >
bq. > When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer)
If you wanted something to only reside in the on heap cache, you'd return 0 for getSerializedLength, and return null for getDeserializer.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 44
bq. > <https://reviews.apache.org/r/1214/diff/25/?file=34983#file34983line44>
bq. >
bq. > Is it ok that "already serialize" and "can't be serialized" return same response out of this method?
Theres no longer an "already serialized" state. I now remove the object entirely from the heap. I'll edit the comments.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 49
bq. > <https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line49>
bq. >
bq. > Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do?
Its a programming error if we do, we shouldn't ever do it, the sizer should make sure of that. I'll add that to the comments.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 52
bq. > <https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line52>
bq. >
bq. > Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava?
Its made by Guava's MapMaker. I'll add it to the comments.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1653
-----------------------------------------------------------
On 2011-08-24 04:08:40, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 04:08:40)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087479#comment-13087479 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-19 03:05:29.951253)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Cacheable interface now far less confusing. HFileBlock has reformatting reverted.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
CHANGES.txt 763ddbc
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073087#comment-13073087 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > conf/hbase-env.sh, line 42
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42>
bq. >
bq. > Extra whitespace is obvious on review board.
Fixed.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2>
bq. >
bq. > Year of copyright.
Fixed.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48>
bq. >
bq. > Long line should wrap.
Fixed.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64>
bq. >
bq. > Indentation should be corrected for the if block.
Fixed
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2>
bq. >
bq. > Copyright missing.
Fixed
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25>
bq. >
bq. > This implies that an eviction may involve fewer than one block.
bq. > If so, this count can be named evictedBlockCount.
Fixed.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>
bq. >
bq. > Should we add hitCount for offHeapCache ?
bq. > This can be addressed in a follow-on JIRA.
I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115>
bq. >
bq. > In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts.
Agreed. I may be redoing this substantially in a follow on Jira.
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145>
bq. >
bq. > Is this a typo ?
Yup, and fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1220
-----------------------------------------------------------
On 2011-07-28 23:02:50, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-28 23:02:50)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072716#comment-13072716 ]
Li Pi commented on HBASE-4027:
------------------------------
The bug actually broke an existing test case. I'm not sure why I didn't notice - I did after trying to rebuild it this morning.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067842#comment-13067842 ]
Li Pi commented on HBASE-4027:
------------------------------
whoops. I have no idea why thats still commented out.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089989#comment-13089989 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-24 03:57:16.993844)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
addressed ted yu's reviews above.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087941#comment-13087941 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-19 20:21:35.572967)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Forgot to override .equals when redoing HFileBlock. Fixed, tests now work.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089850#comment-13089850 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-23 23:02:05.583625)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed race condition. rebased on top of another patch. addressed some of jgray's comments.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091627#comment-13091627 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1653
-----------------------------------------------------------
Ship it!
Excellent. Minor queries from a know-nothing below. If you have a mo, make a few answers. You'll need to make a fat release note in the issue so folks get a clue on what this sweet addition is about and how to run it (release note should be good enough one of us can make a short paragraph on this new facility). Good on you Li.
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3691>
Is it ok that "already serialize" and "can't be serialized" return same response out of this method?
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
<https://reviews.apache.org/r/1214/#comment3690>
When would we have an instance of Cacheable where we did not need to serialize AND deserialize? (Wondering if worth having this extra CacheableDeserializer)
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3692>
Whats this mean ('an exception will be thrown..') Will we ever try to put something into a block that is larger than the block? Its a programming error if we do? If a serialized object is larger than a 'block', what do we do?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3693>
Where is this ConcurrentLinkedHashMap? Its not in this class? I see a ConcurrentMap. Its made in the depths of guava?
- Michael
On 2011-08-24 04:08:40, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 04:08:40)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068513#comment-13068513 ]
stack commented on HBASE-4027:
------------------------------
bq. When we catch a block, we throw it into on-heap.
At one point does stuff go into the offheap cache?
bq. I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option.
How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size'
bq. Apologies for the first few patches.
Don't apologize. This back and forth is healthy.
Looks like the bulk of my comments above can be addressed by comment in code.
Let us know when you think it ready for new review.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087260#comment-13087260 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1525
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3476>
Whitespace.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3477>
Whitespace.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3478>
Auto-formatter weirdness.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3480>
this and all other formatting fixes should probably be in a seperate jira. filed.
- Li
On 2011-08-18 08:31:59, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-18 08:31:59)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073090#comment-13073090 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-07-30 00:39:48.920025)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Addressed TedYu/JGray's comments.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088038#comment-13088038 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1573
-----------------------------------------------------------
Ship it!
All tests passed based on hbase-4027v12.7.diff
- Ted
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067993#comment-13067993 ]
stack commented on HBASE-4027:
------------------------------
Patch looks excellent!
You want that change to surefire heap size in pom.xml? Are you saying here that tests won't pass unless you have 5G of RAM?
In apache code base, you cannot have '@author' tags.
WTF does DoubleBlockCache dooo? (smile). Usually folks will fill in the class comment on the what/why a class exists. Would be good here; e.g. why is it called DoubleBlockCache? (Similar for other new classes you've added)
I love this:
{code}
+ private LruBlockCache onHeapCache;
+ private SlabCache offHeapCache;
{code}
Thats sweet.
Remove code rather than comment it out: e.g. + // onHeapCache.cacheBlock(blockName, buf);
And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here).
Any chance of some offheap stats (is this getStats used?)
We only return heapsize of onheap cache. You think we should not include offheap?
Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context?
Do the assign when you declare the blockSize and slabs data members rather than wait till the constructor runs. Make them final while you are at it. Similar comment for SingleSizeCache, etc.
What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab?
Should slabsize just be hardcoded as Integer.MAX_VALUE?
The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is.
Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs.
You think this is going to happen or how could it happen?
{code}
+ if(backingMap.containsKey(blockName)){
+ throw new RuntimeException("Cached an already cached block");
+ }
{code}
Whats going on here (I see this in a few places):
{code}
+ } catch (Exception e) {
+
+ }
{code}
The get from backingStore will never return a null here?
+ contentBlock = backingStore.get(key).getBlock(key, caching);
Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?
There are some copy paste issues with the javadoc you have in your TestSingleSlabCache.
Hey Li, you get an F in documentation (smile) but the code is great. Good stuff.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092040#comment-13092040 ]
Hudson commented on HBASE-4027:
-------------------------------
Integrated in HBase-TRUNK #2147 (See [https://builds.apache.org/job/HBase-TRUNK/2147/])
HBASE-4027 Enable direct byte buffers LruBlockCache
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/conf/hbase-env.sh
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: HBase-4027 (1).pdf
New updated benchmark results. Tested with 1gb dataset, as per todd's suggestion. the FS cache does better, but still can't match the off heap cache.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081902#comment-13081902 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-09 20:39:55.719673)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Fixed compilation error. Corrected package location of cachetestutils.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
CHANGES.txt e9c0478
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087743#comment-13087743 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1549
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3524>
The following 10 lines of code is very similar to those starting line 65.
Should be able to write another method so that both of them can call.
- Ted
On 2011-08-19 04:16:44, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 04:16:44)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt 0478003
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072472#comment-13072472 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-07-28 18:29:02.934543)
Review request for hbase and Li Pi.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: HBase-4027.pdf
PDF with benchmark results and some design notes.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082693#comment-13082693 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1392
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
<https://reviews.apache.org/r/1214/#comment3179>
I believe this was a typo.
You meant 1 (for bytes), right ?
- Ted
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv3.diff
fixed a couple of the bugs mentioned, will add documentation and do more extensive testing in another diff.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084311#comment-13084311 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-12 17:26:40, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>
bq. >
bq. > this can race against getBlock() though:
bq. >
bq. > Thread A: backingMap.get(key) returns object
bq. > Thread B: put() returns same object
bq. > Thread B: free(object)
bq. > Thread A: use object. boom?
bq. >
bq. > putIfAbsent shouldn't be any slower than put, may as well make use of it
Ah, gotcha! I see it now. Fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1423
-----------------------------------------------------------
On 2011-08-12 08:41:37, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 08:41:37)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073097#comment-13073097 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > conf/hbase-env.sh, lines 44-45
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44>
bq. >
bq. > Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly "a bit above the off heap cache size" is)
I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2>
bq. >
bq. > 2011
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1>
bq. >
bq. > license
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41>
bq. >
bq. > whitespace
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5>
bq. >
bq. > class comment
Added
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39>
bq. >
bq. > Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc).
bq. >
bq. > If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks)
It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches?
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66>
bq. >
bq. > longer than 80 chars
fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77>
bq. >
bq. > This seems like a behavior that we may not always want.
bq. >
bq. > If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?)
bq. >
bq. > we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap)
I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115>
bq. >
bq. > this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both.
The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74>
bq. >
bq. > huh?
Just a typo. Fixed. Don't know why that line is there.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 84
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line84>
bq. >
bq. > line > 80 chars
fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 122-125
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line122>
bq. >
bq. > seems like an odd behavior. this method is kinda non-deterministic from the caller POV, we have no idea whether this passed or failed
If the size of the item being cached is too big to fit into any slab, we can't cache it. Again, I'm moving away from MemCached's model, so this should work better.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 151
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line151>
bq. >
bq. > is this really a cache miss? this cache will never take this block.
Good point. Taking that part out.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 155
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line155>
bq. >
bq. > but here we need to check for null and increment cache miss rather than hit if it is null
Yup, doing so as well.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 170-173
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line170>
bq. >
bq. > this non-javadoc comment is not necessary, just the @Override has been normal format in hbase
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 269
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line269>
bq. >
bq. > should this really be a bunch of INFO logs? Could we have these exported as metrics instead?
Will take care of this next pass.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28135#file28135line1>
bq. >
bq. > license
added.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 374-375
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line374>
bq. >
bq. > formatting looks a little strange here (missing spaces and such)
fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 28
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line28>
bq. >
bq. > This test looks more specific than testing "the concurrent LruBlockCache"
Woah, typo, fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 34
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line34>
bq. >
bq. > you are missing a space here and throughout this file and some others... our format is to always have a space before a {
fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28139#file28139line1>
bq. >
bq. > license
added.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java, line 2
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28138#file28138line2>
bq. >
bq. > 2011
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 376
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28137#file28137line376>
bq. >
bq. > line > 80 chars
Fixed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 63-65
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line63>
bq. >
bq. > Why these ratios? At the least, this should all be configurable (even if just in code and undocumented).
bq. >
bq. > Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive.
bq. >
bq. > I'm okay with the first iteration not being awesome and auto-tuning but this layered cache design seems to make it hard to change anything once it's instantiated.
Its sort of expensive to change cache sizes with DirectByteBuffer after instantiation. I have a plan to restructure this away from MemCached's model though.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 37
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line37>
bq. >
bq. > whitespace here and throughout this file
Removed.
bq. On 2011-07-29 17:41:50, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 48
bq. > <https://reviews.apache.org/r/1214/diff/1/?file=28134#file28134line48>
bq. >
bq. > I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab?
I'm using MemCached's allocation model - although I have one written that does it in a wholly different manner. But MemCached gets around fragmentation by allocating a certain range of sizes to a Single Slab.
If something is smaller than the max size of a particular slab, the extra space is just wasted. Each SingleSizeCache can cache things up to a certain size. Things above that size cannot be cached, and things below that size will waste space.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1211
-----------------------------------------------------------
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054639#comment-13054639 ]
Jonathan Gray commented on HBASE-4027:
--------------------------------------
In the new HFile v2 over in HBASE-3857 the block cache interface changes from ByteBuffer to HeapSize. So you can now put anything you want into the cache that implements HeapSize (there is a new HFileBlock that is used in HFile v2).
One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly. With a DBB you can't get access to an underlying byte[].
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064983#comment-13064983 ]
stack commented on HBASE-4027:
------------------------------
bq. Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though.
Go for it. That should fill your summer Li. Good stuff.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084266#comment-13084266 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1423
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3285>
this can race against getBlock() though:
Thread A: backingMap.get(key) returns object
Thread B: put() returns same object
Thread B: free(object)
Thread A: use object. boom?
putIfAbsent shouldn't be any slower than put, may as well make use of it
- Todd
On 2011-08-12 08:41:37, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 08:41:37)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084003#comment-13084003 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-12 08:41:37.017483)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed test failure, as ted yu reported.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084358#comment-13084358 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-12 20:26:21.230751)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Fixed another broken test case. (Didn't reset buffer position before doing compare) and fixed race.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088089#comment-13088089 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. >
I haven't added in the serializer interface yet. That will be there very very soon - and should allow us to reduce the heapsize by another 60 bytes or so per entry.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1575
-----------------------------------------------------------
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064931#comment-13064931 ]
Li Pi commented on HBASE-4027:
------------------------------
I have a copy based variant working right now, I'll submit the patch by the end of this week for that. Benchmarks show roughly 75% of the speed of the existing LruBlockCache, and around 4x the speed of FS cache reads. Realistically, we'll be using a two stage cache solution, with the LruBlockCache also holding some values, so for the majority of reads, it'll be the same speed as the LruBlockCache.
Currently pursueing a 0 copy solution - going to need to make everything run off ByteBuffers rather than byte[] though.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084264#comment-13084264 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>
bq. >
bq. > > 0, not == 1 -- the contract of compareTo is just that it returns positive, not that it returns exactly 1
bq.
bq. Li Pi wrote:
bq. http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)
bq.
bq. Returns:
bq. -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.
bq.
bq.
bq. Todd Lipcon wrote:
bq. http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)
bq.
bq. "Returns:
bq. a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."
bq.
bq. so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1
Okay. Will fix. Thanks!
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>
bq. >
bq. > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:
bq. >
bq. > ByteBuffer storedBlock = ...allloc
bq. > ... fill it in...
bq. > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
bq. > if (alreadyCached != null) {
bq. > // we didn't insert the new one, so free it and throw an exception
bq. > backingStore.free(storedBlock);
bq. > throw new RuntimeException("already cached xxxxx");
bq. > }
bq. >
bq. > make sense?
bq.
bq. Li Pi wrote:
bq. Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."
bq.
bq. Good change, still though, I changed it to free alreadyCached instead.
bq.
bq. Todd Lipcon wrote:
bq. ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?
Yes, but that someone else stores a reference to the singlesizecache and a copy of the key. They'll look it up by key again, and will hit the new item.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>
bq. >
bq. > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?
bq.
bq. Li Pi wrote:
bq. Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.
bq.
bq. Todd Lipcon wrote:
bq. is that true? wouldn't the 2G offheap always have a subset of what's in heap?
...ah yes, good point. i'll add this check in.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
-----------------------------------------------------------
On 2011-08-12 08:41:37, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 08:41:37)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087233#comment-13087233 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12>
bq. >
bq. > Please add javadoc for these methods.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92>
bq. >
bq. > This shows that Cacheable can reside in on-heap cache.
bq. > The description for Cacheable should be refined.
Done. Cacheable can reside in either.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79>
bq. >
bq. > HeapSize is covered by Cacheable so is not needed here.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528>
bq. >
bq. > Indentation is incorrect here.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535>
bq. >
bq. > Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.
bq. > Please add javadoc.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540>
bq. >
bq. > Please add javadoc for what this method does.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541>
bq. >
bq. > Whitespace.
Done.
bq. On 2011-08-17 04:44:54, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554
bq. > <https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554>
bq. >
bq. > Indentation.
Done.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1489
-----------------------------------------------------------
On 2011-08-18 08:31:59, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-18 08:31:59)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089977#comment-13089977 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 55
bq. > <https://reviews.apache.org/r/1214/diff/17/?file=33818#file33818line55>
bq. >
bq. > does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object.
Fixed. Changed the interface.
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1467
bq. > <https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1467>
bq. >
bq. > this object confuses me. once we've serialized something, all we should need is the ByteBuffer.
Fixed it.
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1518
bq. > <https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1518>
bq. >
bq. > this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual()
Fixed. Comparisons are done with if (this.buf.compareTo(castedComparison.buf) != 0) { instead.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1575
-----------------------------------------------------------
On 2011-08-24 02:46:25, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 02:46:25)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072470#comment-13072470 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-07-28 18:29:54.907234)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Added reviewers. Feel free to review anyways.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: HBase4027v9.diff
configuration changed to percentage of MaxDirectMemorySize.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089059#comment-13089059 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-22 21:55:23.810893)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
CacheableSerializer interface added. HFileBlock now fully serialized.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089948#comment-13089948 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39>
bq. >
bq. > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :)
bq.
bq. Li Pi wrote:
bq. I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way.
We're going to call it Slabs and Blocks. Either way, easy enough change. Will make it consistent.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89>
bq. >
bq. > javadoc
bq.
bq. Li Pi wrote:
bq. Will do.
fixed.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28>
bq. >
bq. > add @param @return javadoc
bq.
bq. Li Pi wrote:
bq. Will do.
Fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1594
-----------------------------------------------------------
On 2011-08-24 02:09:44, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 02:09:44)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089947#comment-13089947 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-24 02:09:44.785814)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
added documentation according to jgrays changes.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089993#comment-13089993 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-24 04:08:40.398370)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed comment.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092070#comment-13092070 ]
Li Pi commented on HBASE-4027:
------------------------------
Huh:
Running org.apache.hadoop.hbase.client.TestAdmin
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
without the patch. Tests don't run?
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068240#comment-13068240 ]
Li Pi commented on HBASE-4027:
------------------------------
-And when we cache a block, how we know where to put it? On or off heap? How you decide where to put it (Looks like you put it off heap always here).
When we catch a block, we throw it into on-heap. Originally, I planned for it to go off heap, but as the cache partitions in LRUBlockCache gives it scan resistance already, this seemed unnecessary.
-Any chance of some offheap stats (is this getStats used?)
getStats now works for the offheap cache, at least for eviction count/cache hit rates.
-We only return heapsize of onheap cache. You think we should not include offheap?
I figured since offheap wasn't really in the heap, I didn't include it. Not sure if this was the best option.
-Yeah, a bunch of these state methods go unimplemented. Can we do any of them? Or is it that they just don't make sense in offheap context?
I took a look, implemented some of them. SlabCache implements everything except getFreeSize. Since all the space it's given is allocated using directbytebuffers immediately, theres never any free size. So I have it return 0.
-What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of 4G per slab?
ByteBuffer positions are addressed using integers. You can only declare one up to 2gb using ByteBuffer.allocateDirect().
-Should slabsize just be hardcoded as Integer.MAX_VALUE?
Sometimes we want a slab thats less than 2gb. Such as when the size for our entire cache is smaller than that.
-The kung fu going on in the shutdown of metaslab needs a comment. I think I know whats going on. Explain what 'cleaner' is.
Yeah, this is how you deallocate directbytebuffers. Added comments, but its still pretty voodoo. Basically cleaner is a deconstructor for a direct byte buffer.
-Is there something up w/ the naming in MetaSlab or is it me misreading? We are iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think we'd be making blocks in a slab, not slabs.
If we want to create N blocks of size X, and a slab, can, at max, contain B blocks, we decrement N by B until N is below B. We create a slab each time and divide that slab into blocks. If N < B and N > 0, then we create our final slab that we use to get space for the remaining blocks.
-You think this is going to happen or how could it happen?
This shouldn't happen unless we have a race condition. But we might, in which case we should throw an exception. I'll add some comments to this portion.
-Whats going on here (I see this in a few places):
If we get a null pointer exception, that means the cache missed. Therefore we return null, increment the missed counter.
-The get from backingStore will never return a null here?
As an invariant, it should never. Because if we are running out of buffers, one should be evicted by the ConcurrentLinkedHashMap when we do a read. On closer inspection, this can happen in a multithreaded environment, I'll figure out a way to fix this. (Probably by synchronizing it.)
-Is this a good name for the default blocksize config? "hbase.mapreduce.hfileoutputformat.blocksize" The 'outputformat' would seem to come from mapreduce world (Which seems to be where you found it if I grep src). SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?
Switched it to DEFAULT_BLOCKSIZE.
Apologies for the first few patches. Apparently I diffed against a different branch than I had intended. Thus the pom.xml edits and commented out code. That was for speed testing and benchmarking. I fixed a few more things, and will continue to search for bugs and add documentation. I'm also implementing better metrics.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086117#comment-13086117 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-17 04:16:06.047294)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
refactored, no more casting, cacheable added. multithreaded test that hammers single key added. general code cleanup stuff as well. heapsize not implemented completely yet.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082713#comment-13082713 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-10 21:28:51, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 21
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21>
bq. >
bq. > I believe this was a typo.
bq. > You meant 1 (for bytes), right ?
Good catch. Fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1392
-----------------------------------------------------------
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072369#comment-13072369 ]
Todd Lipcon commented on HBASE-4027:
------------------------------------
Nice work and nice write-up. One interesting thing I noticed in the benchmarks is that the scalability of the FS cache performance is poor - 10 threads looks like it's only twice as fast as 1 thread on the FS implementations, whereas it scales much better on the various in-process caches. How large is the dataset in this test? Is it small enough that all of the threads are contending to read the same HFile block, and thus hitting the locks in HFile reader?
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054737#comment-13054737 ]
Jason Rutherglen commented on HBASE-4027:
-----------------------------------------
{quote}One big question is whether you're going to make copies out of the direct byte buffers on each read of that block, or if you're going to change KeyValue to use the ByteBuffer interface (or some other) instead of the byte[] directly{quote}
Right the {{HFile.Scanner.getKeyValue()}} method is calling {{block.array()}}. We'd need to track down all {{byte[]}} references, and convert them to {{ByteBuffer}}. That's more of a separate Jira.
I think converting a direct ByteBuffer to byte[] will generate a fair amount of garbage, though of a different (smaller and more numerous) kind than the blocks.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081962#comment-13081962 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
-----------------------------------------------------------
CHANGES.txt
<https://reviews.apache.org/r/1214/#comment3063>
your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment3064>
style:
/**
* ...
*/
public class CacheStates {
(comment formatting and space before '{')
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3068>
hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3069>
does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3065>
hyphenate 'on-heap' and 'off-heap' for clarity
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3066>
missing space - " bytes ..."
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3067>
same as above
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3070>
we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3071>
vertically collapse this - one line per param
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3072>
when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:
ByteBuffer storedBlock = ...allloc
... fill it in...
ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
if (alreadyCached != null) {
// we didn't insert the new one, so free it and throw an exception
backingStore.free(storedBlock);
throw new RuntimeException("already cached xxxxx");
}
make sense?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3073>
I think there's a bug here if you have multiple users hammering the same contentBlock -- two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3074>
this.size() is in units of bytes, not blocks
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3075>
maybe rename to getOccupiedSize?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3076>
wrong Log class - should use org.apache.commons.logging.Log
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3077>
remove extra whitespace
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3078>
hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3079>
LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message -- eg "Unable to deallocate direct memory during shutdown".
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3080>
getBlock*s*Remaining, right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3081>
incomplete comment here
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3083>
this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3085>
this isn't calling addSlabByConf here, so its javadoc seems out of date?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3087>
spelling: proportions
Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3088>
"configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3094>
just to be thorough, check that they aren't negative?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3089>
> 0, not == 1 -- the contract of compareTo is just that it returns positive, not that it returns exactly 1
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3091>
should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3095>
should we also check that they sum up to at least 0.99 or something?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3092>
wrap line
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3093>
you should .trim() the strings so whitespace is ignored
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3096>
is that true? I thought it threw an RTE
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3097>
collapse vertical space
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3098>
this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.
In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3100>
javadoc to indicate where this is called from.
Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:
new SlabItemEvictionWatcher() {
public void onEviction(String key) {
internalBlockEvicted(key);
}
}
that way it doesn't show up as a public API to anyone else
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3102>
I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:
SingleSizeCache.ConcurrentMap.eviction
-> SingleSizeCache's eviction watcher
---> calls stats.evict()
---> removes from backingMap
---> calls SlabCache.onEviction()
-----> calls assignedCache.evict(key)
-------> calls stats.evict() again
-------> removes from map which is always a no-op
right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3103>
may as well just iterate over sizer.values()
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3104>
again can iterate over values()
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3105>
move this function up in the file so it's with other functions. Also needs @Override right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3106>
also move this function up above the inner classes
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3099>
does this make a copy? it shouldn't need to at this point, right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
<https://reviews.apache.org/r/1214/#comment3108>
this can be package-private right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
<https://reviews.apache.org/r/1214/#comment3107>
you don't need abstract inside an interface
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
<https://reviews.apache.org/r/1214/#comment3109>
see note about consistency of conf params above
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
<https://reviews.apache.org/r/1214/#comment3110>
Apache license
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
<https://reviews.apache.org/r/1214/#comment3111>
this is unused?
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
<https://reviews.apache.org/r/1214/#comment3112>
remove empty javadocs
src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java
<https://reviews.apache.org/r/1214/#comment3113>
patch accidentally reverting another recent commit
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3114>
add license
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3115>
long line
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3117>
using a queue here ensures that two threads never request the same block in this test - that's one good test, but there should be another that just caches a small number of blocks and pounds on get only
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3116>
I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:
long total = totalQueries.incrementAndGet();
if (total % 10000 == 0) {
LOG.debug("Ran " + total + " queries");
}
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3118>
rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests -- otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3120>
this class looks like mostly dup code
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment3119>
hrm, this is identical to the other method?
- Todd
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-4027:
--------------------------------------
Release Note: Setting -XX:MaxDirectMemorySize in hbase-env.sh enables this feature. The file already has a line you can uncomment and you need to set the size of the direct memory (your total memory - size allocated to memstores - size allocated to the normal block cache - some head room for the other functionalities). (was: Allows the enabling off an off heap cache in hbase. Set hbase.offheapcachesize in hbase-site.xml
Then, uncomment the line in hbase-env.sh. Cache does a slab allocation model similar to memcached.
)
I changed the release note, the patch doesn't have a "hbase.offheapcachesize" configuration and it's enabled as soon as you set -XX:MaxDirectMemorySize (which is actually a big problem when you consider this: http://hbase.apache.org/book.html#trouble.client.oome.directmemory.leak). I'm opening followup jiras.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091951#comment-13091951 ]
stack commented on HBASE-4027:
------------------------------
Before commit I ran tests.... about 2/3rds completed. TestAdmin failed but figured that not because of this patch.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087232#comment-13087232 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>
bq. >
bq. > I think there's a bug here if you have multiple users hammering the same contentBlock -- two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000
bq.
bq. Li Pi wrote:
bq. changed it to returnBlock.put(contentBlock.duplicate()) instead.
bq.
bq. Todd Lipcon wrote:
bq. don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?
Since we duplicate it, we don't need to rewind after using it. The initial copy is already rewound, so we just duplicate it, do our put, and then discard it. TestHammerSingleKey has been written.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>
bq. >
bq. > hrm, this is identical to the other method?
bq.
bq. Li Pi wrote:
bq. Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.
bq.
bq. Todd Lipcon wrote:
bq. but you should be able to extract a method, even if the method has to take a type parameter. too much dup code
We have a Cacheable interface now. No more dup code.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
-----------------------------------------------------------
On 2011-08-18 08:31:59, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-18 08:31:59)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082987#comment-13082987 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > CHANGES.txt, line 197
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197>
bq. >
bq. > your diff appears to revert this change. perhaps you need to rebase on trunk before you take diff against it.
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines 24-26
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24>
bq. >
bq. > style:
bq. > /**
bq. > * ...
bq. > */
bq. > public class CacheStates {
bq. >
bq. > (comment formatting and space before '{')
Fixed
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 58
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58>
bq. >
bq. > hrm, is this constructor ever meant to be used? If the off-heap cache isn't configured, then it should just instantiate LruBlockCache directly, no?
This is currently taken care of in StoreFile.java, but, good idea, I'll change that.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>
bq. >
bq. > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?
Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 65
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65>
bq. >
bq. > hyphenate 'on-heap' and 'off-heap' for clarity
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>
bq. >
bq. > missing space - " bytes ..."
The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 137
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137>
bq. >
bq. > we should add in the heap size used by the accounting and hashmaps in the off-heap cache as well.
Will take care of next revision.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>
bq. >
bq. > I think there's a bug here if you have multiple users hammering the same contentBlock -- two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000
changed it to returnBlock.put(contentBlock.duplicate()) instead.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 176
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176>
bq. >
bq. > this is a little gross, since the APIs imply that you can cache anything, but the implementation only supports HFileBlock.
bq. >
bq. > In the HFile code review I'd suggested adding a new interface like CacheableBlock that would expose any APIs you need. We should consider doing that now?
Definitely. Will do. At that point, might also change DoubleBlockCache to not cache things that aren't CacheableBlocks.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 190
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190>
bq. >
bq. > maybe rename to getOccupiedSize?
Done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 148
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148>
bq. >
bq. > this.size() is in units of bytes, not blocks
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>
bq. >
bq. > hrm, this is identical to the other method?
Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>
bq. >
bq. > vertically collapse this - one line per param
80 char limit?. I don't know why eclipse keeps doing this. Will fix!
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>
bq. >
bq. > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:
bq. >
bq. > ByteBuffer storedBlock = ...allloc
bq. > ... fill it in...
bq. > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
bq. > if (alreadyCached != null) {
bq. > // we didn't insert the new one, so free it and throw an exception
bq. > backingStore.free(storedBlock);
bq. > throw new RuntimeException("already cached xxxxx");
bq. > }
bq. >
bq. > make sense?
Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."
Good change, still though, I changed it to free alreadyCached instead.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 26
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line26>
bq. >
bq. > wrong Log class - should use org.apache.commons.logging.Log
fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 35
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line35>
bq. >
bq. > remove extra whitespace
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 37-38
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line37>
bq. >
bq. > hm, we have 4 different terms for these: buffers, items, chunks, and blocks. Can we have a terminology that's used consistently throughout?
Okay. Will figure out names.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, lines 90-91
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line90>
bq. >
bq. > LOG.warn("Shutdown failed!", e); is probably what you want. Also improve the text of this error message -- eg "Unable to deallocate direct memory during shutdown".
done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 104
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line104>
bq. >
bq. > getBlock*s*Remaining, right?
Done
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 111
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31772#file31772line111>
bq. >
bq. > incomplete comment here
I have no idea what that comment is supposed to be, removed it.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 88
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line88>
bq. >
bq. > this isn't calling addSlabByConf here, so its javadoc seems out of date?
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 93
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line93>
bq. >
bq. > spelling: proportions
bq. >
bq. > Also make the confs more heirarchical - hbase.offheapcache.slab.sizes or something is easier to parse. If we have any other configs throughout, they should all share a prefix
Fixed. All prefixes changed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 43
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line43>
bq. >
bq. > this class looks like mostly dup code
It is, but SingleSizeCache still caches bytebuffers rather than HFileBlocks.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 42
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line42>
bq. >
bq. > rather than changing the implementation of equals() in HFileBlock, I think it's better to add a static method like blocksContainSameData() here in the tests -- otherwise it might cause problems elsewhere where we were relying on identity-equality for HFileBlock.equals
equals was never implemented beforehand. I added it. I think default equals is just ==? I don't think anyone was using .equals beforehand.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line1>
bq. >
bq. > add license
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java, line 30
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31778#file31778line30>
bq. >
bq. > patch accidentally reverting another recent commit
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 38
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line38>
bq. >
bq. > I think this amount of output is going to severaly limit the throughput at which the threads can pound on the cache. Perhaps below:
bq. > long total = totalQueries.incrementAndGet();
bq. > if (total % 10000 == 0) {
bq. > LOG.debug("Ran " + total + " queries");
bq. > }
I just removed the printout - I don't think its needed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java, line 29
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31779#file31779line29>
bq. >
bq. > long line
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 100
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line100>
bq. >
bq. > "configuration mismatch" is vague for users to understand. Better to say something like "hbase.offheapslabproportions specifies proportions for 4 slab classes, whereas hbase.offheapslabsizes specifies sizes for 5 slab classes." so they can see specifically what the issue is.
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 110
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line110>
bq. >
bq. > just to be thorough, check that they aren't negative?
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>
bq. >
bq. > > 0, not == 1 -- the contract of compareTo is just that it returns positive, not that it returns exactly 1
http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)
Returns:
-1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 116
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line116>
bq. >
bq. > should include the name of the config here too - "Sum of all proportions specified in hbase.blahblah is greater than 1."
Done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 117
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line117>
bq. >
bq. > should we also check that they sum up to at least 0.99 or something?
added a LOG.warn if it doesn't add up to 0.99
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 121
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line121>
bq. >
bq. > wrap line
Done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 143
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line143>
bq. >
bq. > you should .trim() the strings so whitespace is ignored
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 159
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line159>
bq. >
bq. > is that true? I thought it threw an RTE
Yes, it does throw a RTE. Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, lines 161-164
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line161>
bq. >
bq. > collapse vertical space
Done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, lines 375-376
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31776#file31776line375>
bq. >
bq. > see note about consistency of conf params above
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 30
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line30>
bq. >
bq. > you don't need abstract inside an interface
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 23
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31774#file31774line23>
bq. >
bq. > this can be package-private right?
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 231
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line231>
bq. >
bq. > javadoc to indicate where this is called from.
bq. >
bq. > Since this is only used as the callback for the per-size caches, I think it would be better to hide it by making it private, and not having this class implement SlabItemEvictionWatcher. Then when you instantiate the subclasses, pass them an anonymous class:
bq. > new SlabItemEvictionWatcher() {
bq. > public void onEviction(String key) {
bq. > internalBlockEvicted(key);
bq. > }
bq. > }
bq. >
bq. >
bq. > that way it doesn't show up as a public API to anyone else
this is now package private.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 60
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line60>
bq. >
bq. > this needs to also make it a daemon thread, right? You could also look into org.apache.hadoop.hbase.Chore
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 237
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line237>
bq. >
bq. > I think that the assignedCache's evict() counter is going to be double incremented here... call stack like:
bq. >
bq. > SingleSizeCache.ConcurrentMap.eviction
bq. > -> SingleSizeCache's eviction watcher
bq. > ---> calls stats.evict()
bq. > ---> removes from backingMap
bq. > ---> calls SlabCache.onEviction()
bq. > -----> calls assignedCache.evict(key)
bq. > -------> calls stats.evict() again
bq. > -------> removes from map which is always a no-op
bq. >
bq. > right?
Whoops. Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 245
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line245>
bq. >
bq. > may as well just iterate over sizer.values()
fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 315
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line315>
bq. >
bq. > again can iterate over values()
fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 1
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line1>
bq. >
bq. > Apache license
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>
bq. >
bq. > this is unused?
Not anymore
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, lines 59-63
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line59>
bq. >
bq. > remove empty javadocs
Done.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
-----------------------------------------------------------
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v11.7.diff
fixed two typos.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v15.2.diff
The latest patch from reviewboard.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v11.diff
fixed things as per ted/todd's reviews
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072446#comment-13072446 ]
Jonathan Gray commented on HBASE-4027:
--------------------------------------
Has the diff on reviewboard been updated? Trying to review but looks like it's still the old one?
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068539#comment-13068539 ]
Li Pi commented on HBASE-4027:
------------------------------
|At one point does stuff go into the offheap cache?
Err, when we catch a block, it gets thrown into both on-heap and off-heap.
|How would an operator monitor offheap if no stats. What should they do? Its kinda tough because the cache interface only allows return of one 'size'
Thats an unfortunate problem, seeing as the off heap cache is technically made of many differently sized caches. I'll have them log info the same was as the on heap cache does every once in a while.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4027:
-------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thank you for the fat feature Li. Thanks to all the reviewers too. Committed to TRUNK.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087518#comment-13087518 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-19 04:10:41, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49
bq. > <https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49>
bq. >
bq. > If self is always returned, why do we need the return value here ?
it returns a copy of itself, unless it doesn't need deserialization.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1544
-----------------------------------------------------------
On 2011-08-19 04:16:44, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 04:16:44)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt 0478003
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087511#comment-13087511 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1544
-----------------------------------------------------------
Cacheable interface is much more intuitive now.
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3516>
Change an to the.
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3515>
Do I see an incomplete sentence here ?
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3514>
If self is always returned, why do we need the return value here ?
- Ted
On 2011-08-19 03:05:29, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 03:05:29)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. CHANGES.txt 763ddbc
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
bq. src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091254#comment-13091254 ]
Ted Yu commented on HBASE-4027:
-------------------------------
+1 on latest patch.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054216#comment-13054216 ]
Jason Rutherglen commented on HBASE-4027:
-----------------------------------------
Here's the relevant code from hadoop-lzo:
{code}
Object cleaner = buf.getClass().getMethod("cleaner").invoke(buf);
cleaner.getClass().getMethod("clean").invoke(cleaner);
{code}
and
{code}
ByteBuffer.allocateDirect(newSize);
{code}
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv4.5.diff
Fixed as per Todd's reviewboard comments.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081519#comment-13081519 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-09 08:36:02.175711)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Fixed diff. Rebase on 3857 works!
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072424#comment-13072424 ]
Li Pi commented on HBASE-4027:
------------------------------
Dataset is small - I wanted to guarantee it remained in FS cache. I'll make it larger.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027-v10.5.diff
ignore v10 diff, that is just broken. this is the fixed one.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068021#comment-13068021 ]
Li Pi commented on HBASE-4027:
------------------------------
@Ted, if it NPE's, that means that the cache doesn't contain the required item, so it's a cache miss.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067999#comment-13067999 ]
Li Pi commented on HBASE-4027:
------------------------------
Thanks!
Theres quite a few bugs I just discovered. I'm fixing them right now. I'm adding reasonable documentation too now.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Release Note:
Allows the enabling off an off heap cache in hbase. Set hbase.offheapcachesize in hbase-site.xml
Then, uncomment the line in hbase-env.sh. Cache does a slab allocation model similar to memcached.
Status: Patch Available (was: Open)
I'll do the writeup and pretty benchmarked graphs and stuff tomorrow.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086886#comment-13086886 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-18 08:31:59.585101)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
heapsize added. also addressed ted's review above. fixed formatting in hfileblock. (was inconsistent).
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082087#comment-13082087 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1371
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
<https://reviews.apache.org/r/1214/#comment3155>
I think this was intended for use by hbase-env.sh
- Ted
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027v15.3.diff
final one!
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068706#comment-13068706 ]
Ted Yu commented on HBASE-4027:
-------------------------------
In TestSlabCache:
{code}
+ Block[] blocks = generateRandomBlocks(87, blockSize);
{code}
We'd better cover more cases where cache miss is exercised.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082679#comment-13082679 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-10 01:44:22, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line 18
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18>
bq. >
bq. > I think this was intended for use by hbase-env.sh
Yup, it's used by StoreFile.java now. We now specify the size of the off heap cache as a proportion of total direct memory size.
long offHeapCacheSize = (long) (conf.getFloat("hbase.offheapcachepercentage", (float) 0.95) * DirectMemoryUtils.getDirectMemorySize());
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1371
-----------------------------------------------------------
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089063#comment-13089063 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-22 22:01:35.140426)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
added CacheableDeserializer.java.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027v10.6.diff
Moved HFileBlockCacheTestUtils/SingleSizeCacheTestUtils to the tests folder, so it builds under mvn.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072588#comment-13072588 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-07-28 23:02:50.734090)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
small bugfix, off by one error in slab.java. declaring things to maximum size would fail.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070845#comment-13070845 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1191/
-----------------------------------------------------------
Review request for hbase and Todd Lipcon.
Summary
-------
Uploading slabcachepatchv4 to review for Li Pi.
This addresses bug HBASE-4027.
https://issues.apache.org/jira/browse/HBASE-4027
Diffs
-----
conf/hbase-env.sh 2d55d27
pom.xml 729dc37
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION
Diff: https://reviews.apache.org/r/1191/diff
Testing
-------
Thanks,
Todd
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089964#comment-13089964 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-24 02:46:25.061572)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
updated test case.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091965#comment-13091965 ]
Li Pi commented on HBASE-4027:
------------------------------
I'm running a test against the previous version without 4027. I'll let you know what fails.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075966#comment-13075966 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-01 23:54:08.232314)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed build errors. Tested patch/build on fresh pull from trunk.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070862#comment-13070862 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1191/#review1182
-----------------------------------------------------------
could do with some tests for MetaSlab. also some multi-threaded tests - see MultithreadedTestUtil, example usage in TestMemStoreLAB
pom.xml
<https://reviews.apache.org/r/1191/#comment2484>
did you determine that this ConcurrentLinkedHashMap was different than the one in Guava? I thought it got incorporated into Guava, which we already depend on.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2485>
punctuation wise, I think it would be easier to read if you hyphenated on-heap and off-heap. This applies to log messages below as well.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2486>
No need to line-break here
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2487>
consider using StringUtils.humanReadableInt for these sizes.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2488>
@Override
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2489>
when you're just overriding something from the superclass, no need for javadoc unless it says something new and exciting. If you feel like you want to put something there, you can use /** {@inheritDoc} */ to be explicit that you're inheriting from the superclass.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2490>
I think you should only put-back into the on-heap cache in the case that the 'caching' parameter is true.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2491>
hrm, the class javadoc says that the statistics should be cumulative, but this seems to just forward
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2492>
TODOs
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1191/#comment2493>
is this code used? seems like dead copy-paste code to me.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
<https://reviews.apache.org/r/1191/#comment2497>
extraneous debugging left in
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2498>
I think this is usually called a "slab class" - I think that name would be less confusing, since "Meta" is already used in HBase to refer to .META.
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2499>
unclear what the difference is between the two.
Is "slabs" the list of 2GB buffers, and "buffers" is the list of actual items that will be allocated? I think the traditional names here are "slabs" and "items". where each slab holds some number of allocatable items
Also, rather than // comments, use /** javadoc comments */ before the vars
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2500>
these vars probably better called maxBlocksPerSlab and maxSlabSize, since they're upper bounds.
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2501>
I think this code would be a little easier to understand if you split it into one loop for the full slabs, and an if statement for the partially full one. Something like:
int numFullSlabs = numBlocks / maxBlocksPerSlab;
boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) > 0;
for (int i = 0; i < numFullSlabs; i++) {
alloc one of maxSlabSize;
addBuffersForSlab(slab);
}
if (hasPartialSlab) {
alloc the partial one
addBuffersForSlab(slab);
}
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2502>
should be a LOG.warn
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
<https://reviews.apache.org/r/1191/#comment2503>
shouldn't this class have an alloc() and free() method?
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2511>
shouldn't this implement BlockCache?
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2504>
no need to line-break
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2505>
It seems dirty to reach back upwards to "master" here. Cyclical dependencies are a code smell...
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2508>
I don't understand what purpose returnMap is serving here
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2506>
add '@Override', then you don't need to copy-paste javadoc
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2507>
this is where you would call backingStore.alloc() -- then buffers can be made into a private member and keep encapsulation
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2510>
@Override, remove javadoc
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2509>
this is super ugly, plus bad performance - throwing an exception is very expensive, since it has to construct a whole stack trace object, etc.
Instead, check backingMap.get(key) against null
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2512>
why does master need to know about this?
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2495>
numBlocks isn't the size, is it?
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2496>
missing space before "blocks"
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
<https://reviews.apache.org/r/1191/#comment2494>
there are several identical copies of this floating around. Can they all use the same class?
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2513>
overall note: I think we could move all of this to a new io.hfile.slab package
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2520>
should this implement BlockCache?
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2515>
since this is only modified during initialization, we can use an unsynchronized TreeMap which is probably more efficient
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2516>
rename to statThreadPeriodSecs to indicate time unit.
Though, since it's a constant, it probably should be STAT_THREAD_PERIOD. This should be configurable but we can address in a follow-up
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2525>
you can set the thread factory to make daemon threads, then you don't need setDaemon(true) below
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2514>
line breaks unnecessary
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2517>
this stuff will have to be configurable too. but this is fine for now
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2518>
should also note that it returns 0 for the case of an uncacheable object
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2519>
this function is never used -- you should make it private, and then call it from cacheBlock below.
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2521>
again this is no good. check for null instead of catching an exception
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2522>
when would this be true? evicting something that doesn't have an entry in backingStore?
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2523>
it seems like there's a loop in flow control here.
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2524>
move this nice logging output to something like: slabstats.logUsage()
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2526>
since you're using scheduleAtFixedRate, this should implement Runnable, not extend Thread
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2527>
need to be public?
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2528>
should be ALL_CAPS
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2529>
ALSO_ALL_CAPS
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java
<https://reviews.apache.org/r/1191/#comment2530>
why boxed longs?
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
<https://reviews.apache.org/r/1191/#comment2531>
for consistency, I think we should name these something like:
hfile.block.cache.offheap.size
hfile.block.cache.offheap.minblocksize
also, is min block size really a min? isn't it more like expectedblocksize?
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
<https://reviews.apache.org/r/1191/#comment2532>
update this javadoc
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
<https://reviews.apache.org/r/1191/#comment2533>
should use JUnit 4 style tests in new code - i.e not inherit TestCase.
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
<https://reviews.apache.org/r/1191/#comment2535>
nowhere in this test does it actually verify that the data is valid
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
<https://reviews.apache.org/r/1191/#comment2534>
some javadoc would be nice... it seems this code is copy-pasted from TestLruBlockCache, but really doesn't make sense when applied to the SingleSlabCache
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java
<https://reviews.apache.org/r/1191/#comment2536>
see above.
If you can refactor some of this code into something like BlockCacheTestUtils.testBasicCacheContract(BlockCache cache) that would be good... copy paste evil
- Todd
On 2011-07-25 22:55:56, Todd Lipcon wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1191/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-25 22:55:56)
bq.
bq.
bq. Review request for hbase and Todd Lipcon.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Uploading slabcachepatchv4 to review for Li Pi.
bq.
bq.
bq. This addresses bug HBASE-4027.
bq. https://issues.apache.org/jira/browse/HBASE-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. pom.xml 729dc37
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1191/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Todd
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054660#comment-13054660 ]
Li Pi commented on HBASE-4027:
------------------------------
This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073099#comment-13073099 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100
bq. > <https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100>
bq. >
bq. > Should we add hitCount for offHeapCache ?
bq. > This can be addressed in a follow-on JIRA.
bq.
bq. Li Pi wrote:
bq. I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics.
What I meant to say, is, that it does have a hitcount. It just isn't displayed anywhere.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1220
-----------------------------------------------------------
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072684#comment-13072684 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1221
-----------------------------------------------------------
I see you fixed a bug, but you didn't add a new test case! Bug fixes should include regression tests. Also, the new diff changed some formatting - looks like some >80char lines now. Will do a full review tomorrow.
- Todd
On 2011-07-28 23:02:50, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-28 23:02:50)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068544#comment-13068544 ]
Todd Lipcon commented on HBASE-4027:
------------------------------------
In my opinion we need a servlet or jamon page which can dump the stats of the various slab classes as well as their hit/miss rates.
Alternatively, a new MetricsContext for the slab info.
But, somehow, we need to expose information similar to memcached's "stats items" and "stats slabs"
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089990#comment-13089990 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-24 03:46:12, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 52
bq. > <https://reviews.apache.org/r/1214/diff/23/?file=34898#file34898line52>
bq. >
bq. > User may get confused by this javadoc since CacheableDeserializer is interface.
bq. > How about:
bq. > Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer.
fixed.
bq. On 2011-08-24 03:46:12, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 8
bq. > <https://reviews.apache.org/r/1214/diff/23/?file=34899#file34899line8>
bq. >
bq. > This javadoc should be above interface declaration.
fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1612
-----------------------------------------------------------
On 2011-08-24 03:57:16, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 03:57:16)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089134#comment-13089134 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-22 23:09:44.370020)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed concurrency issue. now with readwritelocks.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
CHANGES.txt 0bb7c65
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv4.diff
Added tests for eviction, now logs finely grained stats to file.
Added a bunch of documentation. A bunch - this should take care of most of the documentation concerns.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088015#comment-13088015 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-19 13:30:39, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67
bq. > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67>
bq. >
bq. > Still some white spaces.
fixed.
bq. On 2011-08-19 13:30:39, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244
bq. > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244>
bq. >
bq. > This is an important class related to Cacheable. Should have added javadoc for class and methods.
it's just a pair, added a comment though.
bq. On 2011-08-19 13:30:39, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255
bq. > <https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255>
bq. >
bq. > REFERENCE refers to serializedData, right ?
bq. > I think it would be clearer if the order of second and third parameters gets exchanged.
fixed. added comment.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1546
-----------------------------------------------------------
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054747#comment-13054747 ]
Jason Rutherglen commented on HBASE-4027:
-----------------------------------------
{quote}This would be really useful. I think even making copies out of the direct byte buffers would confer a substantial performance advantage over the FS cache.{quote}
The filesystem cache doesn't help because HBase needs quick access to uncompressed blocks for scanning. For what duration does an uncompressed block need to be cached? In either case, accessing compressed blocks from the FS cache will be faster than hitting the disk or network. I am guessing one can maintain a small'ish block cache, ensure HDFS blocks are local, provide extra space for the FS cache, and gain in read throughput.
Snappy should decompress fast enough for this to be more viable than maintaining a large-ish block cache. The problem [today] with a small'ish block cache is the GC is driven mad.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Priority: Minor
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067930#comment-13067930 ]
Li Pi commented on HBASE-4027:
------------------------------
I'm gonna upload a PDF of benchmark results, but results are interesting. Running HBase standalone, with a small enough dataset so that all files are in local FS cache, the SlabCache substantially outperforms the FS Cache, even if checksumming is disabled.
Compression is disabled as well.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088046#comment-13088046 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1575
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3551>
does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3553>
this object confuses me. once we've serialized something, all we should need is the ByteBuffer.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3555>
this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual()
- Todd
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089994#comment-13089994 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-24 04:00:58, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 10
bq. > <https://reviews.apache.org/r/1214/diff/24/?file=34934#file34934line10>
bq. >
bq. > @return should stay with deserialize() method.
fixed comment.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1614
-----------------------------------------------------------
On 2011-08-24 04:08:40, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 04:08:40)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082998#comment-13082998 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-11 07:52:55.019423)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Fixed as per ted/todd's reviews.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089849#comment-13089849 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java, line 45
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34146#file34146line45>
bq. >
bq. > is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated?
Fixed. Also updated Javadocs in BlockCache.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 27
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34147#file34147line27>
bq. >
bq. > extra newline (maybe move it to below the public class).
bq. >
bq. > also missing a space after CacheStats
fixed.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 55
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34151#file34151line55>
bq. >
bq. > looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor
fixed.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39>
bq. >
bq. > i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :)
I'm probably gonna change this to the MemCached model, where the slabs are "pages", and blocks are "chunks". Or we can just go our own way.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89>
bq. >
bq. > javadoc
Will do.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28>
bq. >
bq. > add @param @return javadoc
Will do.
bq. On 2011-08-22 23:32:16, Jonathan Gray wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 93
bq. > <https://reviews.apache.org/r/1214/diff/19/?file=34152#file34152line93>
bq. >
bq. > should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members
Where should I declare statics? Moving it to the bottom would work.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1594
-----------------------------------------------------------
On 2011-08-23 23:02:05, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-23 23:02:05)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072682#comment-13072682 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1220
-----------------------------------------------------------
conf/hbase-env.sh
<https://reviews.apache.org/r/1214/#comment2739>
Extra whitespace is obvious on review board.
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment2738>
Year of copyright.
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment2740>
Long line should wrap.
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
<https://reviews.apache.org/r/1214/#comment2741>
Indentation should be corrected for the if block.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment2743>
Copyright missing.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment2744>
This implies that an eviction may involve fewer than one block.
If so, this count can be named evictedBlockCount.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2745>
Should we add hitCount for offHeapCache ?
This can be addressed in a follow-on JIRA.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2746>
In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment2747>
Is this a typo ?
- Ted
On 2011-07-28 23:02:50, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-28 23:02:50)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068011#comment-13068011 ]
Li Pi commented on HBASE-4027:
------------------------------
@Ted Yu - thats also a bug. Apparently, I diffed against a testing branch of this patch. Whoops. This explains the commented out code.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv3.2.diff
added some documentation, will add more.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087517#comment-13087517 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-19 04:16:44.682053)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
fixed as per ted yu's comments above.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
CHANGES.txt 0478003
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068582#comment-13068582 ]
stack commented on HBASE-4027:
------------------------------
@Li Related is HBASE-4057?
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081498#comment-13081498 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
-----------------------------------------------------------
(Updated 2011-08-09 07:52:09.666709)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
Changes
-------
Rebased onto 3857. Configuration added as well.
Summary
-------
Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-----
CHANGES.txt 7c6f592
conf/hbase-env.sh 2d55d27
pom.xml 0f24681
src/docbkx/book.xml 2c19cef
src/docbkx/configuration.xml 3595e76
src/docbkx/developer.xml a3e22ea
src/docbkx/performance.xml d8e104f
src/docbkx/troubleshooting.xml 9f93cd9
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7
src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c
src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241
src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937
src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410
src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34
src/main/ruby/hbase/admin.rb 4460d6e
src/site/xdoc/index.xml 8e1b531
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90
src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7
Diff: https://reviews.apache.org/r/1214/diff
Testing
-------
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
Thanks,
Li
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027v6.diff
small bugfix.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089148#comment-13089148 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1594
-----------------------------------------------------------
Looking good! Still some lines > 80 chars and whitespace to fix, but otherwise not much to comment on. As we discussed, I find the Cacheable interface name a bit confusing. Maybe rename it, or just fill the classes and methods with awesome javadoc that explains how it all works :)
Also, should probably add a section to the book about this? Some of these parameters are pretty important and the defaults could be wasteful for some of the default use cases (for example, always giving 20% of heap to blocks > 1.1X block size)
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
<https://reviews.apache.org/r/1214/#comment3591>
is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated?
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
<https://reviews.apache.org/r/1214/#comment3592>
extra newline (maybe move it to below the public class).
also missing a space after CacheStats
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3594>
looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3595>
should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
<https://reviews.apache.org/r/1214/#comment3596>
is this necessary since we throw a runtime exception? could this leave the RS in a weird state? should we actually halt or abort? or will that happen when this gets thrown?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
<https://reviews.apache.org/r/1214/#comment3599>
i think this is the only place you use the term "chunks". should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :)
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
<https://reviews.apache.org/r/1214/#comment3597>
javadoc
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
<https://reviews.apache.org/r/1214/#comment3600>
add @param @return javadoc
- Jonathan
On 2011-08-22 23:09:44, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-22 23:09:44)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt 0bb7c65
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086134#comment-13086134 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1489
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
<https://reviews.apache.org/r/1214/#comment3402>
Please add javadoc for these methods.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
<https://reviews.apache.org/r/1214/#comment3403>
This shows that Cacheable can reside in on-heap cache.
The description for Cacheable should be refined.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3404>
HeapSize is covered by Cacheable so is not needed here.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3405>
Indentation is incorrect here.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3406>
Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means.
Please add javadoc.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3407>
Please add javadoc for what this method does.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3408>
Whitespace.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
<https://reviews.apache.org/r/1214/#comment3409>
Indentation.
- Ted
On 2011-08-17 04:16:06, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-17 04:16:06)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase4027v12.diff
a few more changes, substantially re-factored.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067992#comment-13067992 ]
Ted Yu commented on HBASE-4027:
-------------------------------
Why is the handling below different from that in cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) ?
{code}
public void cacheBlock(String blockName, ByteBuffer buf) {
// onHeapCache.cacheBlock(blockName, buf);
{code}
Overall, more javadoc would help others grasp the functionality for each class.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070859#comment-13070859 ]
stack commented on HBASE-4027:
------------------------------
Doc is great.
These could be final:
+ private LruBlockCache onHeapCache;
+ private SlabCache offHeapCache;
Says 'Metrics are the combined size and hits and misses of both caches' but down in getStats we seem to be getting onheap stats only. Intentional? Same for heapSize.
Do you want to leave this line in hfile? + LOG.debug("decompressedSize = " + decompressedSize);
Whats it mean when you say 'An exception will be thrown if the cached data is larger than the size of the allocated block'?
More notes later.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096353#comment-13096353 ]
Hudson commented on HBASE-4027:
-------------------------------
Integrated in HBase-TRUNK #2172 (See [https://builds.apache.org/job/HBase-TRUNK/2172/])
HBASE-4027 Off Heap Cache never creates Slabs
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089992#comment-13089992 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1614
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
<https://reviews.apache.org/r/1214/#comment3647>
@return should stay with deserialize() method.
- Ted
On 2011-08-24 03:57:16, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-24 03:57:16)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv2.diff
fixed disabled on heap cache.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082835#comment-13082835 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-07-30 02:57:26, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41
bq. > <https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41>
bq. >
bq. > I don't see SkipList being used in this class.
Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing.
bq. On 2011-07-30 02:57:26, Ted Yu wrote:
bq. > conf/hbase-env.sh, line 40
bq. > <https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40>
bq. >
bq. > Still some whitespaces.
Fixed.
bq. On 2011-07-30 02:57:26, Ted Yu wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31
bq. > <https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31>
bq. >
bq. > This doesn't match the actual test.
Fixed.
bq. On 2011-07-30 02:57:26, Ted Yu wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31
bq. > <https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31>
bq. >
bq. > Doesn't match actual name of class.
Fixed.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1238
-----------------------------------------------------------
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. CHANGES.txt e9c0478
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: hbase-4027v13.1.diff
latest version. new cacheableserialization interface.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatchv3.1.diff
fixed test.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084258#comment-13084258 ]
jiraposter@reviews.apache.org commented on HBASE-4027:
------------------------------------------------------
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 64
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64>
bq. >
bq. > does it ever make sense to have offHeapSize < onHeapSize? Perhaps we should have a Preconditions check here?
bq.
bq. Li Pi wrote:
bq. Its useful for testing, I guess? And 8gb heap with 2gb of offheap > 8gb heap without 2gb of offheap.
is that true? wouldn't the 2G offheap always have a subset of what's in heap?
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 67
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67>
bq. >
bq. > missing space - " bytes ..."
bq.
bq. Li Pi wrote:
bq. The string would end up being: "Creating off-heap cache of size 1.9G bytes. Should it be different?
oh, I see - you want it to say "1.9mbytes" instead of "1.9m bytes". That's fine
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 60-67
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line60>
bq. >
bq. > vertically collapse this - one line per param
bq.
bq. Li Pi wrote:
bq. 80 char limit?. I don't know why eclipse keeps doing this. Will fix!
i think it'll still fit in 80 chars without all that vertical space
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 101-103
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101>
bq. >
bq. > when you check up front here, you end up doing two lookups in backingmap. Since this is just a safety check, you could instead check the return value of put() below. Something like:
bq. >
bq. > ByteBuffer storedBlock = ...allloc
bq. > ... fill it in...
bq. > ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
bq. > if (alreadyCached != null) {
bq. > // we didn't insert the new one, so free it and throw an exception
bq. > backingStore.free(storedBlock);
bq. > throw new RuntimeException("already cached xxxxx");
bq. > }
bq. >
bq. > make sense?
bq.
bq. Li Pi wrote:
bq. Doesn't put overwrite the previous value? I guess in this case, it doesn't matter, because you'd just be caching the same thing twice. According to mapmaker: "If the map previously contained a mapping for the key, the old value is replaced by the specified value."
bq.
bq. Good change, still though, I changed it to free alreadyCached instead.
ah, sorry, I meant putIfAbsent from ConcurrentMap. If you free alreadyCached, you might free something that someone's using, right?
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines 118-119
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118>
bq. >
bq. > I think there's a bug here if you have multiple users hammering the same contentBlock -- two people can get to "rewind()" at the same time. You probably need synchronized(contentBlock) around these two lines. See if you can add a unit test which puts just one block in the cache and starts several threads which hammer it - I bet you eventually one of the blocks comes back returned as all 0x0000
bq.
bq. Li Pi wrote:
bq. changed it to returnBlock.put(contentBlock.duplicate()) instead.
don't you need to duplicate it before you call rewind? also, did you add a test that catches this bug, in case there are other similar bugs that I didn't spot?
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 114
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line114>
bq. >
bq. > > 0, not == 1 -- the contract of compareTo is just that it returns positive, not that it returns exactly 1
bq.
bq. Li Pi wrote:
bq. http://download.oracle.com/javase/6/docs/api/java/math/BigDecimal.html#compareTo(java.math.BigDecimal)
bq.
bq. Returns:
bq. -1, 0, or 1 as this BigDecimal is numerically less than, equal to, or greater than val.
bq.
http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Comparable.html#compareTo(java.lang.Object)
"Returns:
a negative integer, zero, or a positive integer as this object is less than, equal to, or greater than the specified object."
so even though sun's BigDecimal happens to return exactly 1, the convention is to check compareTo(...) > 0, because the usual interface is pos/zero/negative, not 1/0/-1
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. > src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java, line 45
bq. > <https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45>
bq. >
bq. > hrm, this is identical to the other method?
bq.
bq. Li Pi wrote:
bq. Ones for SingleSizeCache, which takes ByteBuffers, the other is for a BlockCache, which (hypothetically) takes anything with HeapSize.
but you should be able to extract a method, even if the method has to take a type parameter. too much dup code
- Todd
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
-----------------------------------------------------------
On 2011-08-12 08:41:37, Li Pi wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 08:41:37)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. conf/hbase-env.sh 2d55d27
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers
LruBlockCache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4027:
-------------------------
Attachment: slabcachepatch.diff
Slab allocated cache for hbase. 80% of the speed of the original lrublockcache.
> Enable direct byte buffers LruBlockCache
> ----------------------------------------
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
> Issue Type: Improvement
> Reporter: Jason Rutherglen
> Assignee: Li Pi
> Priority: Minor
> Attachments: slabcachepatch.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside of the heap.
> They need to be manually free'd, which can be accomplished using an documented {{clean}} method.
> The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira