You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2011/12/10 00:52:39 UTC
[jira] [Created] (HBASE-5001) Improve the performance of block
cache keys
Improve the performance of block cache keys
-------------------------------------------
Key: HBASE-5001
URL: https://issues.apache.org/jira/browse/HBASE-5001
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Minor
Fix For: 0.94.0
Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
{quote}
"IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
java.lang.Thread.State: RUNNABLE
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
{quote}
Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167716#comment-13167716 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Whoa... Would not have expected that. Cool. Maybe with this change we're getting into the realm where shrinking the KeyValues would have a measurable effect too.
I'll do the CacheKey thing then.
In fact I would do that even if there was no performance improvement, because it is cleaner.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172676#comment-13172676 ]
stack commented on HBASE-5001:
------------------------------
Or, look, Sebastiano and crew made fastutils apache licensed: http://fastutil.dsi.unimi.it/ Has stuff like http://fastutil.dsi.unimi.it/docs/it/unimi/dsi/fastutil/longs/Long2ObjectMap.html (though this not concurrent... would have to sync it)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5001:
---------------------------------
Status: Open (was: Patch Available)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167344#comment-13167344 ]
stack commented on HBASE-5001:
------------------------------
Hmm... maybe filename compare first.... where filename is byte array, not String?
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172556#comment-13172556 ]
Zhihong Yu commented on HBASE-5001:
-----------------------------------
bq. a bit of sizing of results could save some resizings
I clearly remembered Todd not favoring mini-optimization :-)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5001) Improve the performance
of block cache keys
Posted by "Lars Hofhansl (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167933#comment-13167933 ]
Lars Hofhansl edited comment on HBASE-5001 at 12/12/11 11:14 PM:
-----------------------------------------------------------------
Here's a patch against trunk.
Wasn't as bad as I thought. Made a bunch of things nicer (all the evict by prefix huh-hah is gone, was just used to evict by hfileName)
I ran all the tests that I needed to modify. Will now get a full run.
was (Author: lhofhansl):
Here's a patch against trunk.
Wasn't as bad as I though. Made a bunch of things nicer (all the evict by prefix huh-hah is gone, was just used to evict by hfileName)
I ran all the tests that I needed to modify. Will now get a full run.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5001:
---------------------------------
Attachment: 5001-v1.txt
Here's a patch against trunk.
Wasn't as bad as I though. Made a bunch of things nicer (all the evict by prefix huh-hah is gone, was just used to evict by hfileName)
I ran all the tests that I needed to modify. Will now get a full run.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167690#comment-13167690 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
FWIW, on my unscientific test using the 0.183us solution first described by Lars I'm able to juice out a few thousand more reads per second, usually I'm doing a sustained 255-265k/sec and with it I do 260-270k/sec.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168914#comment-13168914 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
As for the other problem you found... There will be at least one autoboxing from long to Long needed in getLockEntry(...).
Could safe the unboxing in releaseLockEntry(...).
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167284#comment-13167284 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
* Bytes.add(hfileNameInBytes, Bytes.toBytes(offset)) -> 0.07us
But byte[] cannot be directly use as key in a map, no? Would need to wrap in HashBytes, so:
* new HashedBytes(Bytes.add(x, Bytes.toBytes(offser))); -> 0.08us
Which brought me to a new idea, what if we have a CacheKey Object that takes a String and a long:
* new CacheKey(hfileName, offset) -> 0.01us
That would be the cleanest design anyway. Cachkey would implement the proper equals and hashCode methods.
The LruCache could just take CacheKey (or even just java.lang.Object) as cache key, that way we can pass whatever.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167358#comment-13167358 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Was thinking something as simple as this:
{code}
public class Key {
private String file;
private long offset;
public Key(String file, long offset) {
this.file = file;
this.offset = offset;
}
@Override
public int hashCode() {
return file.hashCode()*127+(int)(offset ^ (offset >>> 32));
}
@Override
public boolean equals(Object o) {
if (o instanceof Key) {
Key k = (Key)o;
return offset == k.offset && (file == null ? k.file == null : file.equals(k.file));
} else {
return false;
}
}
}
{code}
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168636#comment-13168636 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
+1, I did a few BC-only tests and it seems to get us a few extra rows per second.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171476#comment-13171476 ]
Hudson commented on HBASE-5001:
-------------------------------
Integrated in HBase-0.92-security #40 (See [https://builds.apache.org/job/HBase-0.92-security/40/])
HBASE-5001 Improve the performance of block cache keys
HBASE-5001 Improve the performance of block cache keys
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemActionWatcher.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167658#comment-13167658 ]
stack commented on HBASE-5001:
------------------------------
LGTM
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169120#comment-13169120 ]
Hudson commented on HBASE-5001:
-------------------------------
Integrated in HBase-TRUNK-security #31 (See [https://builds.apache.org/job/HBase-TRUNK-security/31/])
HBASE-5001 Improve the performance of block cache keys (Lars H and J-D)
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemActionWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167739#comment-13167739 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Actually the CacheKey part would be a bigger change. There's a bunch of code evicting by prefix, grouping by column family, etc.
I think JGray mentioned once that the LruCache is also used outside of HBase at FB.
Should I still make the change, or are we good with the "new StringBuilder(hfileName).append(CACHE_KEY_SEPARATOR).append(offset).toString()" option? (it needs the separator unless we also refactor the caching code)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168718#comment-13168718 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
Alright a proper test now that I removed that machine (also I forgot that I was playing with block sizes, setting them to 16k instead of 4k like I did in the first test gave me a big performance hit).
Without any patch from this jira I can sustain 235-245k/sec.
With the 0.92 patch I can do 245-255k.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167237#comment-13167237 ]
stack commented on HBASE-5001:
------------------------------
What happens if you hfile name is bytes only and we make key by doing Bytes.add, adding the hfilename as bytes + offset as bytes (Would have to include the Bytes.toBytes(long).
We'd have to be doing a lot of key making for the above to show, agreed.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169096#comment-13169096 ]
Hudson commented on HBASE-5001:
-------------------------------
Integrated in HBase-TRUNK #2544 (See [https://builds.apache.org/job/HBase-TRUNK/2544/])
HBASE-5001 Improve the performance of block cache keys (Lars H and J-D)
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemActionWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168651#comment-13168651 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
Stack is pegging one of my test machines (with a constant load fortunately), so I had to first retest the old patch and then did the new one. The difference I saw was it went from 215-225k to 220-230k. I'll do the test with only the other 14 machines to make sure.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171084#comment-13171084 ]
stack commented on HBASE-5001:
------------------------------
I've been profiling to try and figure the slow down scanning in 0.92 that J-D reported up on list and its crazy how much cpu this key making is responsible for in the admittedly warped view the profiler gives you. Let me try this patch on 0.92.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5001:
---------------------------------
Attachment: 5001-0.92.txt
Here's an 0.92 patch.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171229#comment-13171229 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Cool :)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172592#comment-13172592 ]
Todd Lipcon commented on HBASE-5001:
------------------------------------
haha :) My feeling is mostly that profilers point you in the right direction, but it's still worth doing a more macro-benchmark to know that you're really making a difference with the "fixes". But I'm all for shaving off percents here and there - they do add up over time!
Regarding IdLock, we could probably replace it with something like TIntHashSet from trove4j to avoid boxing costs... though that is LGPL so we'd need to do a cleanroom implementation or find a different implementation of similar.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167025#comment-13167025 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Did a quick micro-benchmark:
* hfileName + CACHE_KEY_SEPARATOR + offset -> 0.34us
* new StringBuilder(hfileName).append(CACHE_KEY_SEPARATOR).append(offset).toString() -> 0.19us
* new StringBuilder(hfileName).append(offset).toString() -> 0.183us
StringBuilder(String) reserves an extra 16 characters, so as long as HFiles are < 1PB, this works without extra allocations, and we can save a lot.
One the other hand I have a hard time believing that fractions of microseconds per cache block access would make a difference.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5001:
---------------------------------
Attachment: 5001-v2.txt
Fixed comments and license.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168738#comment-13168738 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Cool... I can commit this to trunk. It's a nice cleanup and a moderate performance improvement (~4%).
I don't see a strong reason for putting this into 0.92.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172546#comment-13172546 ]
stack commented on HBASE-5001:
------------------------------
It does caveat (saving Todd having to intercede), 'Profilers give you distorted view')
Here's a few:
+ util.IdLock releaseLockEntry and getLockEntry take a long and then autobox to check Map. The autoboxing shows as 8% (could keep the Long around down inside IdLock and save '4%'):
3.9% - 3,255 kB - 2,322 alloc. org.apache.hadoop.hbase.util.IdLock.releaseLockEntry
3.7% - 3,035 kB - 2,208 alloc. org.apache.hadoop.hbase.util.IdLock.getLockEntry
+ The put and remove from HBaseServer call queue looks expensive in profiler as does keeping up our MVCC read/write points. Would be sweet to redo these w/ some kinda lockless datastructure (lmax disruptor?). The call queue shows as 20% if you add put and take.
HBaseServer this.callQueue = new LinkedBlockingQueue<Call>(maxQueueSize);
There are some allocation savings we could do too... a bit of sizing of results could save some resizings. E.g.:
2.2% - 13,042 ms org.apache.hadoop.hbase.util.ByteBufferOutputStream.checkSizeAndGrow
If you want access to jprofiler, key just say. jprofiler donated us a license for committers.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167969#comment-13167969 ]
stack commented on HBASE-5001:
------------------------------
Looks great. BlockCacheKey is missing license and class javadoc.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168017#comment-13168017 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
Would you be so kind as to provide a 0.92 patch? My current testing setup is on that version. We could also backport it to 0.92 in a minor release if we see it's stable enough (or even 0.92.0, but I wouldn't go so far).
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-5001:
------------------------------------
Assignee: Lars Hofhansl
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171234#comment-13171234 ]
Hudson commented on HBASE-5001:
-------------------------------
Integrated in HBase-0.92 #192 (See [https://builds.apache.org/job/HBase-0.92/192/])
HBASE-5001 Improve the performance of block cache keys
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemActionWatcher.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168742#comment-13168742 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
Yeah let's keep the changes to 0.92 to a minimum if Stack is going to do the RC today, +1 on commit to trunk. Thanks for the nice work Lars!
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209159#comment-13209159 ]
Phabricator commented on HBASE-5001:
------------------------------------
mbautin has committed the revision "[jira] [HBASE-5001] [89-fb] Improve the performance of block cache keys (J-D, Lars H; pulling from trunk) ".
REVISION DETAIL
https://reviews.facebook.net/D1587
COMMIT
https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1241679
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168644#comment-13168644 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
So it's only marginally faster than the first test you did (with new StringBuilder(hfileName).append(offset).toString())?
Do you have any numbers comparing to the 255-265k/sec and 260-270k/sec above?
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-5001.
----------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to trunk.
Thanks for finding, reviewing, and testing J-D!
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168128#comment-13168128 ]
Hadoop QA commented on HBASE-5001:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507106/5001-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -152 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 75 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/494//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/494//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/494//console
This message is automatically generated.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167669#comment-13167669 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
I agree that the speedup is small, but my test was really done on a purely in-BC dataset :P
Something else that I've just witnessed that's minor is that we're doing a lot of auto-boxing while we could just keep the Long (instead of long) around in IdLock.Entry:
{quote}
java.lang.Thread.State: RUNNABLE
at java.lang.Long.valueOf(Long.java:557)
at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:71)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:250)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
{quote}
and
{quote}
java.lang.Thread.State: RUNNABLE
at java.lang.Long.valueOf(Long.java:557)
at org.apache.hadoop.hbase.util.IdLock.releaseLockEntry(IdLock.java:110)
- locked <0x000000047ba48ba8> (a org.apache.hadoop.hbase.util.IdLock$Entry)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:283)
{quote}
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167748#comment-13167748 ]
Jean-Daniel Cryans commented on HBASE-5001:
-------------------------------------------
I'd be in favor of the bigger refactor, what I think he said about the LRU code is that it's copied from another apache project (solr IIRC).
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171987#comment-13171987 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Did the profiler show any other low-hanging fruit?
Do you have 0.90 comparisons? (Maybe we can figure out what causes the difference)
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168137#comment-13168137 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
Sure. Would be great if we get a comparison in your cluster.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171427#comment-13171427 ]
Hudson commented on HBASE-5001:
-------------------------------
Integrated in HBase-0.92 #193 (See [https://builds.apache.org/job/HBase-0.92/193/])
HBASE-5001 Improve the performance of block cache keys
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5001:
-------------------------
Fix Version/s: (was: 0.94.0)
0.92.0
Committed to 0.92 too.. .makes big difference in profiler view.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167343#comment-13167343 ]
stack commented on HBASE-5001:
------------------------------
I like 0.01us. Better than 0.34us. +1 on your idea Lars. Could compare offsets first. That should be fast. Then filename.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Priority: Minor
> Fix For: 0.94.0
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5001:
---------------------------------
Status: Patch Available (was: Open)
Getting a test run
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171089#comment-13171089 ]
Lars Hofhansl commented on HBASE-5001:
--------------------------------------
The 0.92 patch is good go if we decide to use that.
We do almost the same in 0.90, though (minus the _ part), so that would not explain the 0.92 slowdown.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-0.92.txt, 5001-v1.txt, 5001-v2.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5001) Improve the performance of block
cache keys
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167947#comment-13167947 ]
Ted Yu commented on HBASE-5001:
-------------------------------
In LruBlockCache, please change the javadoc for cacheKey parameter.
> Improve the performance of block cache keys
> -------------------------------------------
>
> Key: HBASE-5001
> URL: https://issues.apache.org/jira/browse/HBASE-5001
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.4
> Reporter: Jean-Daniel Cryans
> Assignee: Lars Hofhansl
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 5001-v1.txt
>
>
> Doing a pure random read test on data that's 100% block cache, I see that we are spending quite some time in getBlockCacheKey:
> {quote}
> "IPC Server handler 19 on 62023" daemon prio=10 tid=0x00007fe0501ff800 nid=0x6c87 runnable [0x00007fe0577f6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hbase.io.hfile.HFile.getBlockCacheKey(HFile.java:457)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:249)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:536)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekExactly(StoreFileScanner.java:219)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:80)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1689)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2857)
> {quote}
> Since the HFile name size is known and the offset is a long, it should be possible to allocate exactly what we need. Maybe use byte[] as the key and drop the separator too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira