You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2012/11/15 00:00:15 UTC
[jira] [Created] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
stack created HBASE-7162:
----------------------------
Summary: Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
Key: HBASE-7162
URL: https://issues.apache.org/jira/browse/HBASE-7162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.96.0
Reporter: stack
Fix For: 0.96.0
These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497758#comment-13497758 ]
Hadoop QA commented on HBASE-7162:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553590/HBASE-4676-common-and-server-v9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 24 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 27 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3342//console
This message is automatically generated.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497779#comment-13497779 ]
stack commented on HBASE-7162:
------------------------------
That is pretty good Matt. Let me commit tomorrow evening or so. Give this some time to stew in case anyone else wants to take a looksee.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Corgan updated HBASE-7162:
-------------------------------
Status: Patch Available (was: Open)
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497569#comment-13497569 ]
stack commented on HBASE-7162:
------------------------------
So, this issue is to cover committing https://issues.apache.org/jira/secure/attachment/12553580/HBASE-4676-common-and-server-v8.patch from hbase-4676. I'll commit after a day or so to give folks a chance to speak up. This commit should be pretty innocuous. Its just move stuff from hbase-server to hbase-common and then introducing an Interface that KeyValue implements, org.apache.hbase.cell.Cell.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Fix For: 0.96.0
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-7162:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed v10 to trunk. Thanks for the patch Matt. Good on you.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502133#comment-13502133 ]
Lars Hofhansl commented on HBASE-7162:
--------------------------------------
That is to say that now we can go through and replace code like this:
{code}
Bytes.compareTo(this.row, 0, row.length, kv.getBuffer(),
kv.getRowOffset(), kv.getRowLength());
{code}
with this:
{code}
Bytes.compareTo(this.row, 0, row.length, kv.getRowArray(),
kv.getRowOffset(), kv.getRowLength());
{code}
(Same for family and qualifier buffers)
Eventually we'll (hopefully) have very few cases where the entire buffer is actually needed, those need to be tackled one by one then.
I can do a first pass through the code and do that.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501682#comment-13501682 ]
Lars Hofhansl commented on HBASE-7162:
--------------------------------------
The Cell stuff is a cool step into the right direction.
We just came across this today again when profiling scanning when FastDiff is enabled.
Turns out that (in our scenario) 60% of the time was spent is just allocating ByteBuffer to back the decoded KeyValues where it would have been sufficient to just create a KV interface that points to an existing buffer for the key and existing buffer for the value.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Corgan updated HBASE-7162:
-------------------------------
Attachment: HBASE-4676-common-and-server-v8.patch
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502169#comment-13502169 ]
Matt Corgan commented on HBASE-7162:
------------------------------------
We could add a getCurrentCell method to the EncodedSeeker interface and deprecate the getKeyValue, getKeyValueBuffer, and getKeyDeepCopy methods. While deprecated, those methods can be reimplemented like:
{code}
@Override
public ByteBuffer getKeyDeepCopy() {
return KeyValueTool.copyKeyToNewByteBuffer(getCurrentCell());
}
@Override
public KeyValue getKeyValue() {
return KeyValueTool.copyToNewKeyValue(getCurrentCell());
}
@Override
public ByteBuffer getKeyValueBuffer() {
return KeyValueTool.copyToNewByteBuffer(getCurrentCell());
}
{code}
{quote}There's one sticky issue still, where we use the key portion of the KV (i.e. row, CF, qual), that would still need to be assembled on each call.{quote}
There's a method for converting from Cell to only the key portion of KeyValue: KeyValueTool.copyKeyToNewByteBuffer. We'd eventually want to stop doing that copying altogether, but it's a quick fix for the tricky places.
There's also an opportunity to switch some of the lower level Bytes.compareTo calls to more readable method calls, like for a row-only comparison we could call CellComparator.compareRow(cellA, cellB).
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498664#comment-13498664 ]
Hudson commented on HBASE-7162:
-------------------------------
Integrated in HBase-TRUNK #3549 (See [https://builds.apache.org/job/HBase-TRUNK/3549/])
HBASE-7162 Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes (Revision 1410213)
Result = FAILURE
stack :
Files :
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueTestUtil.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueTool.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ArrayUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRange.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRangeTool.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/CollectionUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/IterableUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Strings.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/LoadTestKVGenerator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/RedundantKVGenerator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/Cell.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellComparator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellOutputStream.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellScannerPosition.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellTool.java
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteRange.java
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Strings.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/RedundantKVGenerator.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Corgan updated HBASE-7162:
-------------------------------
Attachment: HBASE-7162-common-and-server-v10.patch
patch v10 fixes a couple findbugs warnings, adds ByteRange tests, rebases on trunk, and fixes the jira number in the patch name
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502071#comment-13502071 ]
Matt Corgan commented on HBASE-7162:
------------------------------------
I've been looking through the read path to see what it would take to extend Cell usage out further towards the client, and I haven't found any obstacles that can't be overcome without some work. There are places that are doing comparisons and such on the raw bytes of the KeyValue which won't work. I believe it was the StoreScanner that holds a reference to the current and previous KV which also won't work, but in that particular case it was just a safety check that could probably be removed. I don't understand the lazy-seek stuff well enough yet...
We just have to go through and start changing the individual uses from KeyValue to Cell. To avoid a major refactoring, in problem areas we can make a quick call to KeyValueTool.fromCell(cell) when we need an immutable KeyValue (which is basically the 60% copying cost you are seeing above), and then work on eliminating those wasteful copying calls over time.
All of it would be simpler if we don't have to pass around custom comparators for ROOT and META tables.
As for scatter/gather collection of values, one thing to keep in mind is that the data block encoding is most effective when keys are long and values are short. If your average value is, say, 8 bytes, the setting up an ByteBuffer[] to hold references to all the value ranges may be overkill since each ByteBuffer probably takes something like 40 bytes of heap.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501701#comment-13501701 ]
Lars Hofhansl commented on HBASE-7162:
--------------------------------------
In this case it was the ByteBuffer allocation + writing the deserialized KV into it (BufferedDataBlockEncoder.BufferedEncodedSeeker.getKeyValueBuffer())
Some scatter/gather type concept would be good here.
What you say makes a lot of sense... And would be a major refactoring it seems.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Corgan reassigned HBASE-7162:
----------------------------------
Assignee: Matt Corgan
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501696#comment-13501696 ]
Enis Soztutar commented on HBASE-7162:
--------------------------------------
bq. Turns out that (in our scenario) 60% of the time was spent is just allocating ByteBuffer to back the decoded KeyValues where it would have been sufficient to just create a KV interface that points to an existing buffer for the key and existing buffer for the value.
Does 60% include just the allocation, or allocation + deserialization/copy. I was also thinking along the lines of keeping just the pointers to the block cache for KV's, and only deserializing them to the socket buffers.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502113#comment-13502113 ]
Lars Hofhansl commented on HBASE-7162:
--------------------------------------
The "problem" is KeyValue to getBuffer, which is used quite frequently. But looking at the useages, most of them just get the buffer to compare the row portion, or the qualifier portion, etc. So I was thinking that a great start would be to add a getRowBuffer, getFamilyBuffer, getQualifierBuffer, along the right offset methods to the interface... Then I noticed that's exactly what you did in the Cell interface. :)
There's one sticky issue still, where we use the key portion of the KV (i.e. row, CF, qual), that would still need to be assembled on each call.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497864#comment-13497864 ]
Hadoop QA commented on HBASE-7162:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553628/HBASE-7162-common-and-server-v10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3343//console
This message is automatically generated.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7162) Prefix Compression - Trie data
block encoding; hbase-common and hbase-server changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498751#comment-13498751 ]
Hudson commented on HBASE-7162:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #264 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/264/])
HBASE-7162 Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes (Revision 1410213)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueTestUtil.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueTool.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ArrayUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRange.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRangeTool.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/CollectionUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/IterableUtils.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Strings.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/LoadTestKVGenerator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/RedundantKVGenerator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/Cell.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellComparator.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellOutputStream.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellScannerPosition.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellTool.java
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteRange.java
* /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Strings.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/KeyValueTestUtil.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/RedundantKVGenerator.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch, HBASE-7162-common-and-server-v10.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7162) Prefix Compression - Trie data block
encoding; hbase-common and hbase-server changes
Posted by "Matt Corgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Corgan updated HBASE-7162:
-------------------------------
Attachment: HBASE-4676-common-and-server-v9.patch
v8 patch accidentally had DataBlockEncoding.PREFIX_TREE in the enum. V9 removes it.
> Prefix Compression - Trie data block encoding; hbase-common and hbase-server changes
> ------------------------------------------------------------------------------------
>
> Key: HBASE-7162
> URL: https://issues.apache.org/jira/browse/HBASE-7162
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.96.0
> Reporter: stack
> Assignee: Matt Corgan
> Fix For: 0.96.0
>
> Attachments: HBASE-4676-common-and-server-v8.patch, HBASE-4676-common-and-server-v9.patch
>
>
> These are the hbase-common and hbase-server changes for hbase-4676 Prefix Compression - Trie data block encoding.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira