[jira] Commented: (HBASE-1127) OOME running randomRead PE

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664343#action_12664343 ] 

stack commented on HBASE-1127:
------------------------------

Been looking at this more.  I can watch the GC doing Full, Full, Full, but our executor thread checking SoftValueMap reference queues is not clearing anything.  Then we OOME.  I tried various things including a thread per instance of BlockFSInputStream just blocked on the reference queue waiting for the GC to add stuff.  Odd is that even in this case, we OOME though we get a bit further.  Changing the interval between when our executor thread runs from 10 seconds to 1 second makes it so the executor now does clearing of reference queues but again its not enough.  We'll OOME at about same place as we do when we have a thread per BlockFSInputStream instance (A thread per instance won't fly so this is good)

I'm going to look at this a little more.  In times of high memory pressure, its as though the GC gives up adding items to reference queues which wouldn't seem to make sense.  Given that we're up against the RC, I am currently thinking that I'll revert to having blockcache on by default and instead let users choose it explicitly (with the checker running every second).  I'll leave it on in catalog tables so meta content has block cache on.



> OOME running randomRead PE
> --------------------------
>
>                 Key: HBASE-1127
>                 URL: https://issues.apache.org/jira/browse/HBASE-1127
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.19.0
>
>
> Blockcache is misbehaving on TRUNK.  Something is broke.  We OOME about 20% into the randomRead test.  Looking at heap, its all soft references.  Instrumenting the referencequeue, we're never clearing full gc'ing.  Something is off.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.