You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/04/29 05:15:11 UTC
[jira] [Created] (HBASE-5898) Consider double-checked locking for
block cache lock
Todd Lipcon created HBASE-5898:
----------------------------------
Summary: Consider double-checked locking for block cache lock
Key: HBASE-5898
URL: https://issues.apache.org/jira/browse/HBASE-5898
Project: HBase
Issue Type: Improvement
Components: performance
Affects Versions: 0.94.1
Reporter: Todd Lipcon
Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266123#comment-13266123 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
bq. This addresses the case where there's a really high cache hit ratio, whereas that one addresses the case where there's a 0% cache hit ratio.
Now that I actually read the patch I see how I was wrong, sorry for the noise.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482856#comment-13482856 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
I'd be still worried about checking the cache multiple times especially when the hit ratio is low.
On the other hand in that case the cost of checking the cache twice is probably eclipsed by the cost of actually loading the block... So maybe it's nothing to worry about.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: (was: 5898-v2.txt)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495676#comment-13495676 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Yeah, just verified that RegionServerMetrics.blockCacheMissCount is driven by CacheStats.missCount, which will be double counted with this patch.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266090#comment-13266090 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
The downside is that for a workload with a lower cache hit ratio, we'll ask the cache twice for every cache miss instead of just once. So we should see what the performance difference is for something like a 10MB cache with 1GB dataset, where the dataset fits in the OS buffer cache.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-5898.
----------------------------------
Resolution: Fixed
Committed to 0.94 and 0.96.
(Removal of those unneeded AtomicLongs will give some additional cycles back)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491781#comment-13491781 ]
stack commented on HBASE-5898:
------------------------------
This patch was difference between a stalling sick cluster and one that worked properly again. I think it has to be committed (off-heap cache should not be in the way of committing this issue).
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495786#comment-13495786 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Any comments? Would like to commit and do a 0.94.3rc.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Open (was: Patch Available)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491166#comment-13491166 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
OK. Somebody who wants this committed should do so. I am +0.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496044#comment-13496044 ]
Hudson commented on HBASE-5898:
-------------------------------
Integrated in HBase-0.94 #582 (See [https://builds.apache.org/job/HBase-0.94/582/])
HBASE-5898 Consider double-checked locking for block cache lock (Todd, Elliot, LarsH) (Revision 1408621)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Open (was: Patch Available)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Patch Available (was: Open)
Arggh... NM. This is the right patch.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491971#comment-13491971 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Wanna commit Stack? :)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485354#comment-13485354 ]
stack commented on HBASE-5898:
------------------------------
Onus is on us to show you the thread dumps where no thread was going to hdfs.... let me see if can dig up one.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-5898:
---------------------------------
Attachment: HBASE-5898-0.patch
We saw this on a production cluster.
A single data block was being requested by all ipc handlers. This caused a great deal of lock contention. Todd's patch fixed the issue and caused reads to go back to a normal duration.
This patch is almost identical to Todd's patch. It makes the second try to get a block from the block cache not count as a miss.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485339#comment-13485339 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
I think we should commit this fix (after checking out the TestStoreFile failure) and investigate Ram's issue in a different jira, these look like two different scenarios.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485328#comment-13485328 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Clearly this can happen when HDFS has a problem. One thread tries to load the block, and if that is delayed due to HDFS all other threads need to queue up and wait.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270801#comment-13270801 ]
stack commented on HBASE-5898:
------------------------------
Can you do a test where much lower hit ratio? Any way to compare the amount of 'work' done by the cache or how latency is effected? I'm wondering if you can use your tool to simulate the below:
{quote}
The downside is that for a workload with a lower cache hit ratio, we'll ask the cache twice for every cache miss instead of just once. So we should see what the performance difference is for something like a 10MB cache with 1GB dataset, where the dataset fits in the OS buffer cache.
{quote}
If you can't, is it something you should make it measure?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Patch Available (was: Open)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Mikhail Bautin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272534#comment-13272534 ]
Mikhail Bautin commented on HBASE-5898:
---------------------------------------
Todd: where do we increment those AtomicLong counters?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488477#comment-13488477 ]
stack commented on HBASE-5898:
------------------------------
The offheap cache is being used by no one -- its unusable in current form -- and doubtful anyone would ever use it or if it even makes sense to use it as currently implemented instead of just having bigger block cache w/ decent GC tuning + file system cache instead. I'd be fine for ripping it out until someone wanted to take it up and finish it off.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5898:
-------------------------
Priority: Critical (was: Major)
Marking this critical. Seems like small change w/ big win. All that is missing is a bit of exercise while under heavy read load.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491732#comment-13491732 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
If nobody is willing to sponsor this (by committing), maybe it should not be committed...?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485348#comment-13485348 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Something like this. I don't think the patch works around or even affects the 2nd issue.
The most likely explanation for the 2nd issue seems to be HDFS slowness:
The block is not in the cache, the first thread tries to load it, and while that is happening all other threads have to (and should) wait.
If there is a temporary network hickup that will take a bit... And it would look exactly like these stack traces, where many threads are queued up behind this lock.
Now that you say it, though... On further consideration I am not sure I buy there even *is* a contention issue here. We either:
* have the block in the cache, in which case we'll return it very quickly.
* do not have the block, in that case we have to load it and all other threads must wait.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271791#comment-13271791 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
bq. Can you do a test where much lower hit ratio?
Here it is with 100MB for a ~10GB working data set:
Before:
total=84.22 MB, free=15.78 MB, max=100 MB, blocks=1288, accesses=5078238, hits=3186246, hitRatio=62.74%, cachingAccesses=5078238, cachingHits=3186246, cachingHitsRatio=62.74%, evictions=11196, evicted=1890704, evictedPerRun=168.8731689453125
real 1m12.487s
user 8m54.470s
sys 0m23.850s
After:
LRU Stats: total=84.09 MB, free=15.91 MB, max=100 MB, blocks=1286, accesses=6967721, hits=3189074, hitRatio=45.76%, cachingAccesses=6967721, cachingHits=3189074, cachingHitsRatio=45.76%, evictions=10876, evicted=1887926, evictedPerRun=173.58642578125
real 1m12.614s
user 9m34.990s
sys 0m21.390s
Again the hit ratio is much worse but in reality the hits were about the same in both cases. Included is the "time" output for those runs, the user cpu is somewhat higher (expected) but the total run time is really the same.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495699#comment-13495699 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
same for metaLoads and blockLoads. Both are atomic longs updated for no reason.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490880#comment-13490880 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Let's commit it, and add to the release notes that this potentially breaks off-heap caching.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5898:
-------------------------
Assignee: Todd Lipcon
Assigning Todd.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484290#comment-13484290 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Instead of checking {{blockReadTryCount != 2}} we could also check useLock, right? It should be {{true}} exactly when {{blockReadTryCount != 2}}.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485356#comment-13485356 ]
ramkrishna.s.vasudevan commented on HBASE-5898:
-----------------------------------------------
bq.@Ram: Which version of HBase?
0.94.0 + few more patches over it.
Going thro the code in getLockId and releaseLockId things seems to be pretty much fine there.
Reg could be an HDFS issue,
If that is the case the thread dump should have something where the code is waiting for the HDFS right? somewhere in fsReader.readBlockData() or similar area where we try to read from HDFS.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496003#comment-13496003 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
TestDrainingServer passes locally. TestShell fails with or without this patch.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482949#comment-13482949 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
That's a good theory :)
I guess the worst case here is the uncontended case with lots of cache misses.
For example a single threaded large scan over data not in the block cache; in that case we'll always needlessly go around twice in that loop.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484544#comment-13484544 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550877/HBASE-5898-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestStoreFile
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3152//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266045#comment-13266045 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
Once this is fixed I think we can close HBASE-5000.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485341#comment-13485341 ]
stack commented on HBASE-5898:
------------------------------
[~lhofhansl] When you say different scenarios, are you thinking a.) Todd saw contention on this lock and made a workaround -- this is what this issue addresses -- meanwhile the other scenario b.), is what JD and a few of us saw that looked like weird jvm-bug-like slowness in this same area and this patch works around that issue (Ram saw slowness too)? I'm a little worried that after this patch goes in, b.) will still happen but will just be harder to trip. Would be coolio if could reproduce and poke at it.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488028#comment-13488028 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
You wanna just commit it? It won't do any harm. I'm just doubtful that it will fix anything.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5898:
-------------------------
Attachment: HBASE-5898-1.patch
Retry
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v2.txt
Figured my confusion. Took the patch from the wrong directory. This version is good now.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480566#comment-13480566 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
[~saint.ack@gmail.com] mentioned today that he saw a problem very similar to this.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492790#comment-13492790 ]
Elliott Clark commented on HBASE-5898:
--------------------------------------
Looks like the patch has gone stale after the big metrics2 commit. I'll get a new version up for trunk in a little bit.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490821#comment-13490821 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
If we want to get people testing it I say we include it right now else not at all for 0.94
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490454#comment-13490454 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
I wonder whether HBASE-6032 helps with this.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482866#comment-13482866 ]
Elliott Clark commented on HBASE-5898:
--------------------------------------
Since it's possible that the thread blocked behind a reader that went to hdfs for the block, I think checking the cache is most likely to be a win.
I have a demo app that hits one block multi-threaded. I'll try and get some concrete numbers up to back up my thoughts.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495694#comment-13495694 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
BTW. It also turns out that HFileReaderVx update an AtomicLong (cacheHits) that is not read anywhere...?! I'll remove that.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484292#comment-13484292 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
I meant it is {{false}} when {{blockReadTryCount != 2}}
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487542#comment-13487542 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
What should we do with this one?
@Ram: you're probably right, on the other hand I cannot see anything wrong with this code (and I looked at it multiple times). The proposed patch will probably not fix the problem you saw.
And from some comment earlier it is doubtful that this patch is even an improvement for the issue that Todd saw. I'd say we should table this (at least for 0.94), until we know what exactly the issue is.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271901#comment-13271901 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
Shouldn't be hard to get that CPU time back -- we can just add an array of cacheline-padded AtomicLongs to the cache. Whenever we add something to the cache, we do {{changeCounters[key.hashCode() % changeCounters.length].getAndIncrement()}}. Then change the code to:
{code}
AtomicLong changeCounter = changeCounters[key.hashCode() % changeCounters.length];
long firstTimeChangeCounter = changeCounter.get();
first time:
try to look up in cache
if found: return it
second time:
take lock:
if changeCounter.get() == firstTimeChangeCounter: it's not in cache
otherwise: look up in cache again
{code}
We'd probably want to cache-pad the AtomicLongs too to avoid false sharing.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-5898:
-------------------------------
Attachment: hbase-5898.txt
Here's a patch which implements the double checked locking. The first attempt to look up the value does so without locking it. If it fails, then it tries to acquire the lock, and checks again. If the re-check fails, then it actually goes to disk. Assuming that going to disk is slow, the extra check of cache should be inconsequential, but this made a big difference in throughput of a read-only in-cache workload.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488011#comment-13488011 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
bq. The most likely explanation for the 2nd issue seems to be HDFS slowness
Improbable. With the patch, the first thread would try to get the block and get stuck, the others would first try to read without the lock, fail, and then would also block. Except that when I applied the patch it resolved the issue.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496192#comment-13496192 ]
Hudson commented on HBASE-5898:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #258 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/258/])
HBASE-5898 Consider double-checked locking for block cache lock (Todd, Elliot, LarsH) (Revision 1408620)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Patch Available (was: Open)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264468#comment-13264468 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
this isn't meant to be a final patch - I'm just proposing the idea and put the patch up to illustrate what I meant. It needs more benchmarking in realistic scenarios to know if it's generally a good idea or not
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488470#comment-13488470 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Offtopic: Is the offheap cache even working currently? If not I'd be in favor of ripping that code out.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495982#comment-13495982 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553266/5898-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 93 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
org.apache.hadoop.hbase.TestDrainingServer
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3321//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v4.txt
Attaching same patch again for HadoopQA
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484734#comment-13484734 ]
ramkrishna.s.vasudevan commented on HBASE-5898:
-----------------------------------------------
We have recently hit this issue.. My major concern here is there is contention only happening right? But here in our case the scan itself did not happen for almost 10 mins?
The thread dump clearly says what was found over in this JIRA.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488069#comment-13488069 ]
ramkrishna.s.vasudevan commented on HBASE-5898:
-----------------------------------------------
I think we can commit this. Surely this will have an impact on performance.
Just before committing can we check if Ted's comment
https://issues.apache.org/jira/browse/HBASE-5898?focusedCommentId=13264470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13264470 could be a problem if offheapcache is enabled.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485313#comment-13485313 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Sounds like a missed notify or a deadlock.
Although looking at the code I do not see how that can happen. The use of notify (vs. notifyAll) seems correct in IdLock since all waiting threads wait for the same condition and only one thread will be able to proceed.
@Ram: Which version of HBase?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266054#comment-13266054 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
Really? I don't think this solves the same problem as HBASE-5000. This addresses the case where there's a really high cache hit ratio, whereas that one addresses the case where there's a 0% cache hit ratio.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266059#comment-13266059 ]
stack commented on HBASE-5898:
------------------------------
@Todd Double-checked locking should "...should usually be avoided." [1].
Looking at block cache, it looks like a block should be fully initialized before its added to the cache so we should avoid the horror stories detailed in the article.
Let me try take it for a run...
1. http://en.wikipedia.org/wiki/Double-checked_locking#Usage_in_Java
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495053#comment-13495053 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553057/5898-v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 87 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3308//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Open (was: Patch Available)
Oops. No, not quite the same.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5898:
------------------------------
Attachment: 5898-TestBlocksRead.txt
Some changes to TestBlocksRead are needed to make it pass.
See if the increase in blocks read is acceptable.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264470#comment-13264470 ]
Zhihong Yu commented on HBASE-5898:
-----------------------------------
Consider the case where off heap cache is enabled.
>From DoubleBlockCache:
Suppose getBlock() is executed without the lock (first pass in the new loop of readBlock) and doesn't find cacheKey from onHeapCache but finds it in offHeapCache - it will call onHeapCache.cacheBlock():
{code}
public Cacheable getBlock(BlockCacheKey cacheKey, boolean caching) {
Cacheable cachedBlock;
if ((cachedBlock = onHeapCache.getBlock(cacheKey, caching)) != null) {
stats.hit(caching);
return cachedBlock;
} else if ((cachedBlock = offHeapCache.getBlock(cacheKey, caching)) != null) {
if (caching) {
onHeapCache.cacheBlock(cacheKey, cachedBlock);
}
{code}
Another thread calls cacheBlock() around the same time and executes onHeapCache.cacheBlock() for the same cacheKey:
{code}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf) {
onHeapCache.cacheBlock(cacheKey, buf);
offHeapCache.cacheBlock(cacheKey, buf);
}
{code}
I think there is a race condition which didn't exist before the proposed change: the entries for the same cacheKey in onHeapCache and offHeapCache would diverge.
If off heap cache is disabled, I don't see problem with proposed optimization.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v2.txt
Last try.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-0.94.txt
0.94 version of the patch
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266149#comment-13266149 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
bq. The downside is that for a workload with a lower cache hit ratio, we'll ask the cache twice for every cache miss instead of just once.
Won't this also skew the cache stats themselves?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495570#comment-13495570 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Actually... Which metric is measures via JMX? It would be bad if that metric would be double reported now via JMX.
This is last jira open against 0.94.3.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484297#comment-13484297 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Otherwise looks good. I think the double cache check is fine.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v3.txt
Version that fixes TestStoreFile (by using getMissCachingCount() instead of getMissCount())
The missCount is still over counted.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Patch Available (was: Open)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495472#comment-13495472 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
OK... Will commit in the next hour.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v2.txt
Rebased trunk patch. (want to get 0.94.3 RC out)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484856#comment-13484856 ]
ramkrishna.s.vasudevan commented on HBASE-5898:
-----------------------------------------------
I can attach some parts of the thread dump
bq.Was it a bunch of threads getting same block?
Yes
bq.The double-checked is probably better anyways but could the issue come back just less frequently after this patch goes in?
Am not sure. We tried to restart the client twice still this persisted. Later the RS we restarted after that we could not get this.
This thing repeats many times. We took 3 thread dumps in a span of 2 mins
{code}
"IPC Server handler 42 on 60020" daemon prio=10 tid=0x00007f2f38f1a000 nid=0x6c4d in Object.wait() [0x00007f2f33e4f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:77)
- locked <0x00000006cc2a7178> (a org.apache.hadoop.hbase.util.IdLock$Entry)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:290)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:523)
- locked <0x000000069a665420> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:399)
- locked <0x000000069a665420> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3424)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3379)
- locked <0x000000069a7da458> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3396)
- locked <0x000000069a7da458> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2411)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
{code}
Also we could see that sometimes the relaseLock was also happening. But in the 3 thread dumps this came only once.
{code}
"IPC Server handler 18 on 60020" daemon prio=10 tid=0x00007f2f38ee9800 nid=0x6c35 runnable [0x00007f2f35667000]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.notify(Native Method)
at org.apache.hadoop.hbase.util.IdLock.releaseLockEntry(IdLock.java:108)
- locked <0x00000006cc2a7178> (a org.apache.hadoop.hbase.util.IdLock$Entry)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:352)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:523)
- locked <0x000000069a89d678> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:399)
- locked <0x000000069a89d678> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3424)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3379)
- locked <0x000000069ae0bbb8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3396)
- locked <0x000000069ae0bbb8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2411)
{code}
All the client threads were hanging here
{code}
"Thread-29" prio=10 tid=0x00007f9f2c549000 nid=0x639f waiting for monitor entry [0x00007f9f2adec000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:955)
- waiting to lock <0x000000078ba82828> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:942)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:131)
{code}
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264456#comment-13264456 ]
Zhihong Yu commented on HBASE-5898:
-----------------------------------
Interesting idea.
Minor comments:
The indentation for while (true) loop is off.
Changes to conf/hbase-site.xml belong to another JIRA.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495919#comment-13495919 ]
stack commented on HBASE-5898:
------------------------------
The repeat is a bit strange but it is at least easy to follow what is going on in this block getting code so +1 on commit for 0.94. No harm in beefing up this comment on commit:
{code}
+ * @param repeat Whether this is a repeat lookup for the same block
+ * {@see HFileReaderV2#readBlock(long, long, boolean, boolean, boolean, BlockType)}
{code}
You refer to the code responsible for this 'repeat' param but maybe make mention of not wanting to double-count metrics when doing double-check locking in the cited code?
Good on you Lars.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495013#comment-13495013 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553048/5898-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 87 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestStoreFile
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3307//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266101#comment-13266101 ]
stack commented on HBASE-5898:
------------------------------
Thanks for the prescription Todd.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266064#comment-13266064 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
bq. @Todd Double-checked locking should "...should usually be avoided." [1].
I am above the law, Stack, didn't you know that? :)
Seriously, though, I think the use of the idiom here is safe.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485155#comment-13485155 ]
stack commented on HBASE-5898:
------------------------------
I've seen that thread dump before! [~eclark] has a program to try and repro the above and hopefully we can add some instrumentation and get clues on why the above happens.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484739#comment-13484739 ]
stack commented on HBASE-5898:
------------------------------
[~ram_krish] How you mean Ram? It was stuck where in particular? Was it a bunch of threads getting same block? What did the thread dump look like? There is some issue in here around the wait/notify it seems as implemented. The double-checked is probably better anyways but could the issue come back just less frequently after this patch goes in?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490784#comment-13490784 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Gentlemen, what should we do with this? :)
I would like the next 0.94RC soon. Do we want this now, or can it wait a few weeks for the next RC?
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: 5898-v4.txt
Patch that handled the cache misses correctly. Also removes the unnecessary AtomicLong updates.
Not pretty... BlockCache.getBlock has another boolean parameter to indicate whether cache misses should be counted... All callers and implementors needed to be changed.
Please let me know what you think.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484549#comment-13484549 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Test failure is suspicious :)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495947#comment-13495947 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Will update the comment. I would like to get a HadoopQA run through, but it's currently broken. (possibly related to HBASE-7104)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Open (was: Patch Available)
I can't make patches.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Mikhail Bautin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272561#comment-13272561 ]
Mikhail Bautin commented on HBASE-5898:
---------------------------------------
Is there any way to avoid messing up cache hit ratio with these optimizations? Maybe pass a flag to getBlock saying we should not update the metrics.
Another suggestion from Kannan: maybe we should move metric updates and other extra logic outside of the lock, and only leave cache lookup and HDFS I/O inside the lock.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-5898:
---------------------------------
Status: Patch Available (was: Open)
patch without the counter.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267005#comment-13267005 ]
Jean-Daniel Cryans commented on HBASE-5898:
-------------------------------------------
A not too scientific comparison.
Before:
total=6.26 GB, free=1.04 GB, max=7.3 GB, blocks=101753, accesses=5078311, hits=4333154, hitRatio=85.32%, cachingAccesses=5078311, cachingHits=4333154, cachingHitsRatio=85.32%, evictions=54, evicted=643404, evictedPerRun=11914.888671875
After:
total=6.45 GB, free=878.07 MB, max=7.3 GB, blocks=104747, accesses=5810869, hits=4345850, hitRatio=74.78%, cachingAccesses=5810869, cachingHits=4345850, cachingHitsRatio=74.78%, evictions=52, evicted=627755, evictedPerRun=12072.2119140625
This is using the benchmark tool I'm working on right now, I had to port Todd's patch since I'm not going through the HFile code but I'm doing exactly what he does. The test is started completely cold. It uses 5 threads.
The big takeout is that it did 14% more cache accesses which had the side effect of lowering the hit ratio. The number of actual cache hits was a big higher.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Fix Version/s: 0.96.0
0.94.3
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272716#comment-13272716 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
Yep, I agree we need to fix the metrics messup before this is a candidate for commit.
I like the idea of moving the metric updates out.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495979#comment-13495979 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553266/5898-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 93 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3320//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Status: Open (was: Patch Available)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-5898:
---------------------------------
Attachment: HBASE-5898-1.patch
Good catch on that. Here's a patch without the counting.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496081#comment-13496081 ]
Hudson commented on HBASE-5898:
-------------------------------
Integrated in HBase-0.94-security #83 (See [https://builds.apache.org/job/HBase-0.94-security/83/])
HBASE-5898 Consider double-checked locking for block cache lock (Todd, Elliot, LarsH) (Revision 1408621)
Result = SUCCESS
larsh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488478#comment-13488478 ]
stack commented on HBASE-5898:
------------------------------
[~eclark] Elliott, you get any more data on the hung up case (Elliott has been trying to repo it so can get more data on slow wait/notifying)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496049#comment-13496049 ]
Hudson commented on HBASE-5898:
-------------------------------
Integrated in HBase-TRUNK #3534 (See [https://builds.apache.org/job/HBase-TRUNK/3534/])
HBASE-5898 Consider double-checked locking for block cache lock (Todd, Elliot, LarsH) (Revision 1408620)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492728#comment-13492728 ]
Hadoop QA commented on HBASE-5898:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552544/HBASE-5898-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3258//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495962#comment-13495962 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Triggered HadoopQA directly through jenkins.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5898:
---------------------------------
Attachment: (was: 5898-v2.txt)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488479#comment-13488479 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
Yeah, offheap caching for diskblocks always seemed a bit "interesting" as the blocks all have the same size the GC should be able to deal with the churn nicely.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495098#comment-13495098 ]
stack commented on HBASE-5898:
------------------------------
Seems fine by me Lars.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, 5898-v3.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495032#comment-13495032 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
TestStoreFile fails because it checks CacheStats.getMissCount(), which will be wrong even after modified patch. CacheStats.miss updates two counters: missCount and missCachingCount. Only missCachingCount is counted correctly.
So, we can change the test and accept the wrong number for missCount, or we change the CacheStats API.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, 5898-v2.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491999#comment-13491999 ]
Lars Hofhansl commented on HBASE-5898:
--------------------------------------
In the light of HBASE-6852, even saving taking the lock if we already have cached the block, might lead to a nice performance improvement.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266665#comment-13266665 ]
stack commented on HBASE-5898:
------------------------------
j-d Can you run your blockcache cache checker against this? That'd probably be the best way of figuring if there a problem here (do you have a verify you got the right block; i.e. do a read from the block?)
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for
block cache lock
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272543#comment-13272543 ]
Todd Lipcon commented on HBASE-5898:
------------------------------------
Mikhail: whenever we insert something into the cache, we'd increment the counter for that hash bucket.
> Consider double-checked locking for block cache lock
> ----------------------------------------------------
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.94.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira