You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Bruno Dumon (JIRA)" <ji...@apache.org> on 2011/01/31 20:28:29 UTC
[jira] Created: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
HMaster sometimes hangs during initialization due to missing notify call
------------------------------------------------------------------------
Key: HBASE-3493
URL: https://issues.apache.org/jira/browse/HBASE-3493
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.0
Environment: Linux, HBase 0.90
Reporter: Bruno Dumon
Attachments: assignment-manager-regions-notify-patch.txt
During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
{code}
this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
{code}
And waitForAssignment does this:
{code}
synchronized(regions) {
while(!regions.containsKey(regionInfo)) {
regions.wait();
}
}
{code}
However, I could not find any call to regions.notify(), so this could wait forever.
I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
For reference, this was based on seeing the following from jstack:
{noformat}
"Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa5196fb8> (a java.util.TreeMap)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
- locked <0xa5196fb8> (a java.util.TreeMap)
at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-3493:
----------------------------
Assignee: Bruno Dumon
> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
> Key: HBASE-3493
> URL: https://issues.apache.org/jira/browse/HBASE-3493
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Environment: Linux, HBase 0.90
> Reporter: Bruno Dumon
> Assignee: Bruno Dumon
> Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
> synchronized(regions) {
> while(!regions.containsKey(regionInfo)) {
> regions.wait();
> }
> }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa5196fb8> (a java.util.TreeMap)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
> - locked <0xa5196fb8> (a java.util.TreeMap)
> at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988870#comment-12988870 ]
Jonathan Gray commented on HBASE-3493:
--------------------------------------
+1. i don't see a notify either
> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
> Key: HBASE-3493
> URL: https://issues.apache.org/jira/browse/HBASE-3493
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Environment: Linux, HBase 0.90
> Reporter: Bruno Dumon
> Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
> synchronized(regions) {
> while(!regions.containsKey(regionInfo)) {
> regions.wait();
> }
> }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa5196fb8> (a java.util.TreeMap)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
> - locked <0xa5196fb8> (a java.util.TreeMap)
> at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989220#comment-12989220 ]
Hudson commented on HBASE-3493:
-------------------------------
Integrated in HBase-TRUNK #1726 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1726/])
> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
> Key: HBASE-3493
> URL: https://issues.apache.org/jira/browse/HBASE-3493
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Environment: Linux, HBase 0.90
> Reporter: Bruno Dumon
> Assignee: Bruno Dumon
> Fix For: 0.90.1
>
> Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
> synchronized(regions) {
> while(!regions.containsKey(regionInfo)) {
> regions.wait();
> }
> }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa5196fb8> (a java.util.TreeMap)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
> - locked <0xa5196fb8> (a java.util.TreeMap)
> at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988866#comment-12988866 ]
stack commented on HBASE-3493:
------------------------------
I don't see a notify either(!) (Hey Bruno).
> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
> Key: HBASE-3493
> URL: https://issues.apache.org/jira/browse/HBASE-3493
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Environment: Linux, HBase 0.90
> Reporter: Bruno Dumon
> Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
> synchronized(regions) {
> while(!regions.containsKey(regionInfo)) {
> regions.wait();
> }
> }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa5196fb8> (a java.util.TreeMap)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
> - locked <0xa5196fb8> (a java.util.TreeMap)
> at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HBASE-3493) HMaster sometimes hangs during
initialization due to missing notify call
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3493.
--------------------------
Resolution: Fixed
Fix Version/s: 0.90.1
Hadoop Flags: [Reviewed]
Committed branch and trunk. Thanks for patch Bruno.
> HMaster sometimes hangs during initialization due to missing notify call
> ------------------------------------------------------------------------
>
> Key: HBASE-3493
> URL: https://issues.apache.org/jira/browse/HBASE-3493
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Environment: Linux, HBase 0.90
> Reporter: Bruno Dumon
> Assignee: Bruno Dumon
> Fix For: 0.90.1
>
> Attachments: assignment-manager-regions-notify-patch.txt
>
>
> During HMaster.finishInitialization(), assignRootAndMeta() is called, which at some point does:
> {code}
> this.assignmentManager.waitForAssignment(HRegionInfo.FIRST_META_REGIONINFO);
> {code}
> And waitForAssignment does this:
> {code}
> synchronized(regions) {
> while(!regions.containsKey(regionInfo)) {
> regions.wait();
> }
> }
> {code}
> However, I could not find any call to regions.notify(), so this could wait forever.
> I have not noticed this problem on a real cluster, only when using HBaseTestingUtility on a slow and low-memory Hudson server (I can reproduce it on my local machine by creating some background load). Adding a notify() call when regions.put() is called seems to fix the problem. Will attach patch.
> For reference, this was based on seeing the following from jstack:
> {noformat}
> "Master:0;lat:44776" prio=10 tid=0x08d5b400 nid=0x381a in Object.wait() [0x9c76d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa5196fb8> (a java.util.TreeMap)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.hbase.master.AssignmentManager.waitForAssignment(AssignmentManager.java:1152)
> - locked <0xa5196fb8> (a java.util.TreeMap)
> at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:440)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira