You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2011/10/12 01:25:11 UTC
[jira] [Created] (HBASE-4577) Region server reports storefileSizeMB
bigger than storefileUncompressedSizeMB
Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
-----------------------------------------------------------------------------
Key: HBASE-4577
URL: https://issues.apache.org/jira/browse/HBASE-4577
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Minor
Fix For: 0.92.0
Minor issue while looking at the RS metrics:
bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
I guess there's a truncation somewhere when it's adding the numbers up.
FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142779#comment-13142779 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
The trunk also failed yesterday:
https://builds.apache.org/job/HBase-0.92/100/testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoadWithSplit/
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142866#comment-13142866 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
Yes,I think so. But I am diging.
Because I am not familiar with MR and this issue is not very important to release 0.92 version, Please give me some time.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142007#comment-13142007 ]
Ted Yu commented on HBASE-4577:
-------------------------------
I don't see 'Too many open files' for
https://builds.apache.org/job/PreCommit-HBASE-Build/133//testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoadWithSplit/
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4577) Region server reports storefileSizeMB
bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
gaojinchao updated HBASE-4577:
------------------------------
Attachment: HBASE-4577_trunk.patch
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143172#comment-13143172 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
I didn't find any exception log. It seems the test case has a bug.
Processing steps:
1. Creating table with 5 regions
2. Producing data base on 5 regions
3. Changing table to 15 regions
4. Loading data to new table
Some cases, the data of 1 region may become the data of 11 regions. but the parameter of hbase.bulkload.retries.number is set to 10,we only try 10 time, some data can't load to the region.
If I am wrong, Please correct me! Thanks
I think we should change table to 14 rather than 15 regions in this case.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142012#comment-13142012 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
My local test result:
Running org.apache.hadoop.hbase.TestMultiVersions
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.045 sec
Results :
Failed tests: testHBaseFsck(org.apache.hadoop.hbase.util.TestHBaseFsck): expected:<0> but was:<1>
Tests in error:
testMasterFailoverWithMockedRITOnDeadRS(org.apache.hadoop.hbase.master.TestMasterFailover): test timed out after 180000 milliseconds
testEnableTableRoundRobinAssignment(org.apache.hadoop.hbase.client.TestAdmin): org.apache.hadoop.hbase.TableNotEnabledException: testEnableTableAssignment
testBadOriginalRootLocation(org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster): unknown host: example.org
Tests run: 1073, Failures: 1, Errors: 3, Skipped: 9
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148948#comment-13148948 ]
Hudson commented on HBASE-4577:
-------------------------------
Integrated in HBase-0.92 #125 (See [https://builds.apache.org/job/HBase-0.92/125/])
HBASE-4577 Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138182#comment-13138182 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
It seems result is ok.
numberOfStores=1, numberOfStorefiles=12, storefileUncompressedSizeMB=2344, storefileSizeMB=2344, compressionRatio=1.0000, memstoreSizeMB=100, storefileIndexSizeMB=1, readRequestsCount=0, writeRequestsCount=35004, rootIndexSizeKB=1045, totalStaticIndexSizeKB=1935, totalStaticBloomSizeKB=0, totalCompactingKVs=5481732, currentCompactedKVs=5132546, compactionProgressPct=0.0, coprocessors=[]
rawDataSize is instead of uncompressedBytes ? all docs use uncompressedBytes.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142015#comment-13142015 ]
Ted Yu commented on HBASE-4577:
-------------------------------
@Jinchao:
Please start with test output of TestHFileOutputFormat#testMRIncrementalLoadWithSplit and see why the test failed.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141941#comment-13141941 ]
Hadoop QA commented on HBASE-4577:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501906/HBASE-4577_trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -165 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/133//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/133//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/133//console
This message is automatically generated.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142187#comment-13142187 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
Sorry, I am not familiar with MR. Continue to dig this issue tomorrow.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4577) Region server reports storefileSizeMB
bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
gaojinchao updated HBASE-4577:
------------------------------
Attachment: HBASE-4577_trial_Trunk.patch
This patch is just that my intentions. I don't test and there may be some problems.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141904#comment-13141904 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
Ok, Thanks.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143174#comment-13143174 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
look this logs:
2011-11-02 04:23:42,761 ERROR [main] mapreduce.LoadIncrementalHFiles(214): Retry attempted 10 times without completing, bailing out
2011-11-02 04:23:42,761 ERROR [main] mapreduce.LoadIncrementalHFiles(240): -------------------------------------------------
Bulk load aborted with some files not yet loaded:
-------------------------------------------------
hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,27.bottom
hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,27.top
hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,28.bottom
hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,28.top
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4577) Region server reports storefileSizeMB
bigger than storefileUncompressedSizeMB
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4577:
-------------------------
Status: Patch Available (was: Open)
Patch looks good to me. Submitting to patch build (In future Gao, be careful and put spaces around operators as in ' + ' rather than '+ ' -- good stuff Gao).
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134252#comment-13134252 ]
Jean-Daniel Cryans commented on HBASE-4577:
-------------------------------------------
I'm not sure what the best solution is, but I think the current one is confusing since both metrics share the same name with only "Uncompressed" being different. Looking at the code it seems to be also confused.
The javadoc for Store.getStoreSizeUncompressed:
bq. The size of the store files, in bytes, uncompressed.
And for Store.getStorefilesSize:
bq. The size of the store files, in bytes.
Even the name of the variable is misleading because it's total data size, not total size:
bq. totalUncompressedBytes
They read the same, but the first one is really the size of the uncompressed data only. I think names should be changed to reflect what they really mean, no need for new metrics.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4577) Region server reports storefileSizeMB
bigger than storefileUncompressedSizeMB
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4577:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed branch and trunk. Thanks for patch Gaojinchao.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135686#comment-13135686 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
All have the same block structure, I think storefileUncompressedSizeMB means sum of all uncompressedSizeWithHeader
Here are some comments
/** Total uncompressed bytes, maybe calculate a compression ratio later. */
protected long totalUncompressedBytes = 0;
I try to make a patch. Can you review whether it makes sense ?
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134013#comment-13134013 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
Hi J-D
It seems that:
storefileUncompressedSizeMB only includes datablock size. but storefileSizeMB is the size of total Hfile. So they is different.
Do we need add other size?
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
gaojinchao reassigned HBASE-4577:
---------------------------------
Assignee: gaojinchao
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133922#comment-13133922 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
I am studying the HFile version 2. Can I handle this issue?
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142796#comment-13142796 ]
stack commented on HBASE-4577:
------------------------------
Gaojinchao, so you think the failure in patch-build unrelated to your change (which seems fair to say)
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "ramkrishna.s.vasudevan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141939#comment-13141939 ]
ramkrishna.s.vasudevan commented on HBASE-4577:
-----------------------------------------------
+1 for commit
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "gaojinchao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142008#comment-13142008 ]
gaojinchao commented on HBASE-4577:
-----------------------------------
Test failed, it seems not a patch problem.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141776#comment-13141776 ]
Jean-Daniel Cryans commented on HBASE-4577:
-------------------------------------------
Looks good to me after some cleanup.
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148980#comment-13148980 ]
Hudson commented on HBASE-4577:
-------------------------------
Integrated in HBase-TRUNK #2430 (See [https://builds.apache.org/job/HBase-TRUNK/2430/])
HBASE-4577 Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports
storefileSizeMB bigger than storefileUncompressedSizeMB
Posted by "Matt Corgan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134408#comment-13134408 ]
Matt Corgan commented on HBASE-4577:
------------------------------------
I think the intention of adding it was to see how big the data would be in memory as opposed to on disk, which is a valuable metric. However, we're already jumping ahead to doing delta encoding and prefix compression, so there will soon be a need for a third metric to track encoded size. Maybe these 3 names would be better:
storefileSize: size as reported by the filesystem (lzo/gzip compressed)
encodedDataSize: size in the block cache (with delta encoding or prefix compression, but no gzip)
rawDataSize (instead of uncompressedBytes): size when stored in the current concatenated KeyValue format (the biggest of the 3)
The last 2 would only count datablocks of KeyValues. I'm not sure where bloomfilters and indexblocks should be counted into these. Possibly separate metrics?
> Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
> -----------------------------------------------------------------------------
>
> Key: HBASE-4577
> URL: https://issues.apache.org/jira/browse/HBASE-4577
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: gaojinchao
> Priority: Minor
> Fix For: 0.92.0
>
>
> Minor issue while looking at the RS metrics:
> bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008
> I guess there's a truncation somewhere when it's adding the numbers up.
> FWIW there's no compression on that table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira