You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/02/24 00:07:49 UTC
[jira] [Created] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Add baseline compression efficiency to DataBlockEncodingTool
------------------------------------------------------------
Key: HBASE-5469
URL: https://issues.apache.org/jira/browse/HBASE-5469
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Minor
DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235272#comment-13235272 ]
Hadoop QA commented on HBASE-5469:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519372/D2409.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1249//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1249//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1249//console
This message is automatically generated.
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5469:
-------------------------------
Attachment: D2409.2.patch
mbautin updated the revision "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
Reviewers: JIRA, dhruba, tedyu, stack, heyongqiang
Addressing Ted's comments.
REVISION DETAIL
https://reviews.facebook.net/D2409
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java
src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5469:
-------------------------------
Attachment: D2409.1.patch
mbautin requested code review of "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
Reviewers: JIRA, dhruba, tedyu, stack
DataBlockEncodingTool currently does not provide baseline compression
efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if
we are using LZO to compress blocks, we would like to have the following
columns in the report (possibly as percentages of raw data size).
Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V
DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on
disk)
Background: we never store compressed blocks in cache, but we always store
encoded data blocks in cache if data block encoding is enabled for the column
family.
TEST PLAN
* Run unit tests.
* Run DataBlockEncodingTool on a variety of real-world HFiles.
REVISION DETAIL
https://reviews.facebook.net/D2409
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java
src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/5403/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236084#comment-13236084 ]
Hadoop QA commented on HBASE-5469:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519504/D2409.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1264//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1264//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1264//console
This message is automatically generated.
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237530#comment-13237530 ]
Hudson commented on HBASE-5469:
-------------------------------
Integrated in HBase-TRUNK #2694 (See [https://builds.apache.org/job/HBase-TRUNK/2694/])
[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool
Summary:
DataBlockEncodingTool currently does not provide baseline compression
efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if
we are using LZO to compress blocks, we would like to have the following
columns in the report (possibly as percentages of raw data size).
Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V
DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on
disk)
Background: we never store compressed blocks in cache, but we always store
encoded data blocks in cache if data block encoding is enabled for the column
family.
This patch also has multiple bugfixes and improvements to DataBlockEncodingTool,
including presentation format, memory requirements (reduced 3x) and fixing the
handling of compression.
Test Plan:
* Run unit tests.
* Run DataBlockEncodingTool on a variety of real-world HFiles.
Reviewers: JIRA, dhruba, tedyu, stack, heyongqiang
Reviewed By: tedyu
Differential Revision: https://reviews.facebook.net/D2409 (Revision 1304626)
Result = SUCCESS
mbautin :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch, jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236019#comment-13236019 ]
Phabricator commented on HBASE-5469:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:183 It is not invalidated anywhere. It is cached here so that we do not encode it twice.
REVISION DETAIL
https://reviews.facebook.net/D2409
BRANCH
add_baseline_compression_efficiency_to_HBASE-5469_v5
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235241#comment-13235241 ]
Phabricator commented on HBASE-5469:
------------------------------------
tedyu has accepted the revision "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
It would be nice to attach sample output from DataBlockEncodingTool to JIRA.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:53 Nice catch.
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:150 Nice.
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:166 This doesn't match the actual parameter name.
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java:183 Where is cachedEncodedData invalidated ?
REVISION DETAIL
https://reviews.facebook.net/D2409
BRANCH
add_baseline_compression_efficiency_to_HBASE-5469_v5
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5469:
----------------------------------
Attachment: jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch
The exact patch that was committed.
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch, jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236042#comment-13236042 ]
Phabricator commented on HBASE-5469:
------------------------------------
mbautin has commented on the revision "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
Ted: thanks for the review!
Everyone else: is this OK to commit? Could someone else +1 too? Thanks!
REVISION DETAIL
https://reviews.facebook.net/D2409
BRANCH
add_baseline_compression_efficiency_to_HBASE-5469_v6
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5469:
----------------------------------
Status: Patch Available (was: Open)
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237193#comment-13237193 ]
Phabricator commented on HBASE-5469:
------------------------------------
mbautin has committed the revision "[jira] [HBASE-5469] Add baseline compression efficiency to DataBlockEncodingTool".
REVISION DETAIL
https://reviews.facebook.net/D2409
COMMIT
https://reviews.facebook.net/rHBASE1304626
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch, jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5469) Add baseline compression efficiency
to DataBlockEncodingTool
Posted by "Mikhail Bautin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin updated HBASE-5469:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk.
> Add baseline compression efficiency to DataBlockEncodingTool
> ------------------------------------------------------------
>
> Key: HBASE-5469
> URL: https://issues.apache.org/jira/browse/HBASE-5469
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: D2409.1.patch, D2409.2.patch, jira-HBASE-5469-Add-baseline-compression-efficiency--2012-03-23_15_04_41.patch
>
>
> DataBlockEncodingTool currently does not provide baseline compression efficiency, e.g. Hadoop compression codec applied to unencoded data. E.g. if we are using LZO to compress blocks, we would like to have the following columns in the report (possibly as percentages of raw data size).
> Baseline K+V in blockcache | Baseline K + V on disk (LZO compressed) | K + V DataBlockEncoded in block cache | K + V DataBlockEncoded + LZOCompressed (on disk)
> Background: we never store compressed blocks in cache, but we always store encoded data blocks in cache if data block encoding is enabled for the column family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira