[jira] Resolved: (HBASE-3493) HMaster sometimes hangs during initialization due to missing notify call

["stack (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-3493.
--------------------------

       Resolution: Fixed
    Fix Version/s: 0.90.1
     Hadoop Flags: [Reviewed]

Committed branch and trunk.  Thanks for patch Bruno.

> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
>                 Key: HBASE-3493
>                 URL: https://issues.apache.org/jira/browse/HBASE-3493
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.90.0
>         Environment: Linux, HBase 0.90
>            Reporter: Bruno Dumon
>            Assignee: Bruno Dumon
>             Fix For: 0.90.1
>
>         Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
>     synchronized(regions) {
>       while(!regions.containsKey(regionInfo)) {
>         regions.wait();
>       }
>     }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xa5196fb8> (a java.util.TreeMap)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
>         - locked <0xa5196fb8> (a java.util.TreeMap)
>         at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
>         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
>         at java.lang.Thread.run(Thread.java:619)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira