You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/09/03 03:03:11 UTC
[jira] [Created] (HBASE-4330) Fix races in slab cache
Fix races in slab cache
-----------------------
Key: HBASE-4330
URL: https://issues.apache.org/jira/browse/HBASE-4330
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.92.0
A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097303#comment-13097303 ]
Li Pi commented on HBASE-4330:
------------------------------
Spoke to todd on these changes. +1. Some races remain, but I'm hunting them down.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105069#comment-13105069 ]
Hudson commented on HBASE-4330:
-------------------------------
Integrated in HBase-TRUNK #2211 (See [https://builds.apache.org/job/HBase-TRUNK/2211/])
HBASE-4330 Fix races in slab cache (Li Pi & Todd)
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330v6.txt
Fixed race condition leading to the test failure.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103673#comment-13103673 ]
Ted Yu commented on HBASE-4330:
-------------------------------
+1 on patch v7.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-4330:
-------------------------------
Attachment: hbase-4330.txt
Some new tests and some fixes, let me know what you think
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Assignee: Li Pi (was: Todd Lipcon)
Status: Patch Available (was: Open)
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330v4.txt
rebased.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099838#comment-13099838 ]
Li Pi commented on HBASE-4330:
------------------------------
The final test, TestCacheMultiThreadedEvictions, took around 30 minutes. Its not hanging - it's just taking its sweet time. I'm investigating this right now.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096577#comment-13096577 ]
Li Pi commented on HBASE-4330:
------------------------------
Do we really need:
contentBlock.serializedData.rewind();?
As we always do a ByteBuffer.duplicate later.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101989#comment-13101989 ]
Li Pi commented on HBASE-4330:
------------------------------
Yeah. This doesn't take care of it completely. I'll figure something out. It
does take care of one case though.
On Sep 9, 2011 9:49 PM, "Ted Yu (JIRA)" <ji...@apache.org> wrote:
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102908#comment-13102908 ]
Li Pi commented on HBASE-4330:
------------------------------
I'm looping mvn on the latest patch - I don't see any failures yet.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105012#comment-13105012 ]
Ted Yu commented on HBASE-4330:
-------------------------------
So far the slab unit tests passed on MacBook and Linux.
Integrated to TRUNK.
Thanks for the patch Li and Todd.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100765#comment-13100765 ]
Li Pi commented on HBASE-4330:
------------------------------
Done.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096579#comment-13096579 ]
Li Pi commented on HBASE-4330:
------------------------------
Also, whats the advantage of using a synchronize block and checking whether eviction has been completed yet vs using a RWL?
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330v7.txt
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100773#comment-13100773 ]
Li Pi commented on HBASE-4330:
------------------------------
Woah, not sure what happened there. Fixing.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330v3.txt
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099602#comment-13099602 ]
Li Pi commented on HBASE-4330:
------------------------------
Ran 3 instances of the tests in a loop for 24 hours+. No errors.
mvn crashed many times, failing to run the tests at all, however. I never managed to get it to hang.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101974#comment-13101974 ]
Ted Yu commented on HBASE-4330:
-------------------------------
I don't see much difference for patch v6 on my MacBook:
{code}
testCacheMultiThreadedEviction(org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache) Time elapsed: 23.649 sec <<< ERROR!
java.lang.RuntimeException: Deferred
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76)
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:97)
at org.apache.hadoop.hbase.io.hfile.CacheTestUtils.hammerEviction(CacheTestUtils.java:211)
at org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache.testCacheMultiThreadedEviction(TestSlabCache.java:87)
...
Caused by: java.lang.RuntimeException: already cached key_8_9
at org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache.cacheBlock(SingleSizeCache.java:132)
at org.apache.hadoop.hbase.io.hfile.slab.SlabCache.cacheBlock(SlabCache.java:207)
at org.apache.hadoop.hbase.io.hfile.CacheTestUtils$3.doAnAction(CacheTestUtils.java:197)
at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:139)
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:115)
{code}
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100769#comment-13100769 ]
Ted Yu commented on HBASE-4330:
-------------------------------
Patch v3 contains way too many changes.
Can you rebase and produce a cleaner patch ?
Thanks
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096573#comment-13096573 ]
Ted Yu commented on HBASE-4330:
-------------------------------
When I ran TestSingleSizeCache, it seemed to hang.
Here is part of jstack:
http://pastebin.com/juL5ezzt
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100971#comment-13100971 ]
Ted Yu commented on HBASE-4330:
-------------------------------
We're getting close.
The three tests passed on Linux.
But on MacBook:
{code}
testCacheMultiThreadedEviction(org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache) Time elapsed: 44.652 sec <<< ERROR!
java.lang.RuntimeException: Deferred
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76)
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:97)
at org.apache.hadoop.hbase.io.hfile.CacheTestUtils.hammerEviction(CacheTestUtils.java:208)
at org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache.testCacheMultiThreadedEviction(TestSlabCache.java:87)
...
Caused by: java.lang.RuntimeException: already cached key_2_3
at org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache.cacheBlock(SingleSizeCache.java:132)
at org.apache.hadoop.hbase.io.hfile.slab.SlabCache.cacheBlock(SlabCache.java:207)
at org.apache.hadoop.hbase.io.hfile.CacheTestUtils$3.doAnAction(CacheTestUtils.java:194)
at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:139)
at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:115)
{code}
Here is OS info:
{code}
Darwin tyumac.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 i386
{code}
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097311#comment-13097311 ]
Todd Lipcon commented on HBASE-4330:
------------------------------------
Let's not commit til we get all of them figured out. No sense having 5 jiras all entitled "Fix races in slab cache" :)
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103010#comment-13103010 ]
Li Pi commented on HBASE-4330:
------------------------------
The change in v7 that fixes things:
- scache.cacheBlock(blockName, cachedItem); // if this
- // fails, due to
- // block already
- // being there, exception will be thrown
- backingStore.put(blockName, scache);
+
+ /*This will throw a runtime exception if we try to cache the same value twice*/
+ scache.cacheBlock(blockName, cachedItem);
+
+ /*Spinlock, if we're spinlocking, that means an eviction hasn't taken place yet*/
+ while (backingStore.putIfAbsent(blockName, scache) != null) {
+ Thread.yield();
+ }
The test failed when the following occurred:
Invariant:
Both SSC and SC have the same contents:
Violation:
Item A is in both SSC and SC.
Thread A: evicts A from ssc.
Thread B: starts doing a put into thread SC, -
Thread B: gets directed into SSC, starts doing put in SSC,
Thread B: put goes through thanks to the occuring eviction.
Thread A: calls evictor on SC, removing the object from SlabCache.
result: Object is in SSC, but not SC.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100789#comment-13100789 ]
Ted Yu commented on HBASE-4330:
-------------------------------
{code}
Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failure or timeout
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15 minutes 5 seconds
{code}
Here is the jstack: http://pastebin.com/vDCBMyrq
Here is the OS:
{code}
Linux us01.ciq.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
{code}
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100866#comment-13100866 ]
Li Pi commented on HBASE-4330:
------------------------------
This should fix the problem as reported in your stack trace. A non daemon thread was never ended, therefore the test never completed.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4330:
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt, hbase-4330v6.txt, hbase-4330v7.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100740#comment-13100740 ]
Ted Yu commented on HBASE-4330:
-------------------------------
Please fix the following:
{code}
[INFO] Compilation failure
/home/hadoop/hbase/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java:[47,33] unreported exception java.lang.InterruptedException; must be caught or declared to be thrown
/home/hadoop/hbase/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java:[67,33] unreported exception java.lang.InterruptedException; must be caught or declared to be thrown
{code}
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099610#comment-13099610 ]
Ted Yu commented on HBASE-4330:
-------------------------------
How long did TestSingleSizeCache take, on average ?
Thanks
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097767#comment-13097767 ]
Li Pi commented on HBASE-4330:
------------------------------
Okay :)
@Ted Yu - I can't recreate the hanging TestSingleSizeCache. Every once a while mvn fails to run any tests after looping, but I've determined thats mvn being mvn.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330v5.txt
shuts down the scheduledThreadPool now
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt, hbase-4330v3.txt, hbase-4330v4.txt, hbase-4330v5.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4330) Fix races in slab cache
Posted by "Li Pi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Pi updated HBASE-4330:
-------------------------
Attachment: hbase-4330.txt
fixed evictor resource starvation. Removed spinlock.
Spinlock, with enough threads, was starving the evictionthread of cycles. This causes the tests to run extremely slowly, giving the appearance of a hang.
> Fix races in slab cache
> -----------------------
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt, hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and proposed fixes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira