You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mingjian (JIRA)" <ji...@apache.org> on 2011/07/18 03:48:00 UTC
[jira] [Created] (HBASE-4111) readBlock in HFile.Reader not 64 KB,
but 5~6MB
readBlock in HFile.Reader not 64 KB, but 5~6MB
----------------------------------------------
Key: HBASE-4111
URL: https://issues.apache.org/jira/browse/HBASE-4111
Project: HBase
Issue Type: Bug
Components: io, regionserver
Reporter: mingjian
Assignee: mingjian
Attachments: code and result.txt
We found a strange problem in our read test.
It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067401#comment-13067401 ]
Li Pi commented on HBASE-4111:
------------------------------
I'll take care of this anyways. Should do it before I submit my patch. Otherwise, mine won't work so well.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068705#comment-13068705 ]
Li Pi commented on HBASE-4111:
------------------------------
Can't seem to reproduce this bug using YCSB. I'm running HBase in Standalone mode, dumping in a bunch of records with.
java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -load -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloadc -p columnfamily=family -p recordcount=1000000 -s > load.dat.
No matter how low I turn the cache, or turn it off, my log looks like:
2011-07-20 16:14:28,879 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65678
2011-07-20 16:14:28,880 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65678
2011-07-20 16:14:28,881 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65678
2011-07-20 16:14:28,882 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65608
2011-07-20 16:14:28,883 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65678
2011-07-20 16:14:28,884 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65552
2011-07-20 16:14:28,884 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65578
2011-07-20 16:14:28,887 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65678
2011-07-20 16:14:28,887 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65653
2011-07-20 16:14:28,888 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65658
2011-07-20 16:14:28,889 DEBUG org.apache.hadoop.hbase.io.hfile.HFile: decompressedSize = 65542
....
YCSB records are generally around the same size though - do you have any large entries?
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068816#comment-13068816 ]
stack commented on HBASE-4111:
------------------------------
@mingjian Can you help us? Li can't reproduce. Can you figure what might different about your environment? Thanks.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4111) readBlock in HFile.Reader not 64 KB,
but 5~6MB
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4111:
----------------------------
Attachment: code and result.txt
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068682#comment-13068682 ]
Li Pi commented on HBASE-4111:
------------------------------
mingjian - what happens if you disable the block cache completely?
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4111) readBlock in HFile.Reader not 64 KB,
but 5~6MB
Posted by "mingjian (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingjian updated HBASE-4111:
----------------------------
Description:
We found a strange problem in our read test.
It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
Why not 64 KB? The btrace code and results are in the attachments.
was:
We found a strange problem in our read test.
It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
Why not 64 KB? The btrace code and results are in the attachments.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: mingjian
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4111) readBlock in HFile.Reader not 64 KB,
but 5~6MB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-4111:
----------------------------
Assignee: Li Pi (was: mingjian)
Assigning this to you Li, not because I'm saying you should fix this, but just so you know this thing exists; you are messing in this area and you might trip over it or know whats up here.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068707#comment-13068707 ]
stack commented on HBASE-4111:
------------------------------
Good on you Li Pi.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4111) readBlock in HFile.Reader not 64
KB, but 5~6MB
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068752#comment-13068752 ]
Li Pi commented on HBASE-4111:
------------------------------
Ran complete test.
2011-07-20 18:16:10,070 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=79.46 MB, free=19.29 MB, max=98.75 MB, blocks=1255, accesses=5085601, hits=2477127, hitRatio=48.70%%, cachingAccesses=5085597, cachingHits=2477127, cachingHitsRatio=48.70%%, evictions=16501, evicted=2607215, evictedPerRun=158.00344848632812
CacheSize set to 0.1 of heapsize. Made sure it was hitting disk for most of writes.
Bug can't be reproduced. Though my testload merely has a bunch of very small files.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4111) readBlock in HFile.Reader not 64 KB,
but 5~6MB
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi resolved HBASE-4111.
--------------------------
Resolution: Not A Problem
Not an issue. Heapsize was 50mb, so when cache was set to 0.10, the Index Block could not be Cached.
Thus, indexblock was being repeatedly decompressed.
> readBlock in HFile.Reader not 64 KB, but 5~6MB
> ----------------------------------------------
>
> Key: HBASE-4111
> URL: https://issues.apache.org/jira/browse/HBASE-4111
> Project: HBase
> Issue Type: Bug
> Components: io, regionserver
> Reporter: mingjian
> Assignee: Li Pi
> Attachments: code and result.txt
>
>
> We found a strange problem in our read test.
> It is a 5 nodes cluster.Four of our 5 regionservers set "hfile.block.cache.size"=0.4, one of them is 0.1(we call it node A). When we random read from a 2TB data table we found node A's network reached 100MB, and others are less than 10MB. So the read speed is low.
> We set node A's "hfile.block.cache.size"=0.2, then all the nodes's network are 10MB, that's right. To find why is this we debug with btrace and find "readBlock" in HFile.Reader become abnormal.We know hbase read a block which is 64 KB from disks and put it into blockcache. But when we set "hfile.block.cache.size"=0.1, it is not 64KB, it is 5~6MB one time after about 1 minute we restart hbase.
> Why not 64 KB? The btrace code and results are in the attachments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira