You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "HBase Review Board (JIRA)" <ji...@apache.org> on 2010/10/01 23:28:33 UTC
[jira] Commented: (HBASE-3068) IllegalStateException when new
server comes online, is given 200 regions to open and 200th region gets
timed out of regions in transition
[ https://issues.apache.org/jira/browse/HBASE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917043#action_12917043 ]
HBase Review Board commented on HBASE-3068:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/930/
-----------------------------------------------------------
Review request for hbase and Jonathan Gray.
Summary
-------
Fix is two-fold.
First, added new facility where on successful open, we go and update the timers on all regions in transition that were on the same server.
Secondly, in the timeout monitor, we'll do necessary cleanup and state transitions so that when we go into re-assign, we have the proper state
M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
Changed regionPlans to be a concurrentskiplist. Makes life easier
and in no place do we need lock on regionPlans to span other than
regionPlans changes.
Added to the processing of successful region open, the cleanup
of its regionPlan and a run of updateTimers.
Put setOffline in place of some code that duplicated what it did.
This addresses bug hbase-3068.
http://issues.apache.org/jira/browse/hbase-3068
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1003330
Diff: http://review.cloudera.org/r/930/diff
Testing
-------
Basic unit tests seem to be passing. Testing now up on cluster.
Thanks,
stack
> IllegalStateException when new server comes online, is given 200 regions to open and 200th region gets timed out of regions in transition
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3068
> URL: https://issues.apache.org/jira/browse/HBASE-3068
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
>
> Yesterday we committed a change that makes it so the master will crash is a zk transition that is unexpected. Its extreme but good for highlighting bad state changes (we also started marking these as illegalstateexceptions yesterday too).
> So, testing new master I brought up a new server. Balancer tried to give new server 256 regions.
> {code}
> 2010-10-01 16:01:42,972 INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance in 0ms. Moving 256 regions off of 7 overloaded servers onto 1 less loaded servers
> {code}
> Turns out we failed complete open of all 256 servers within the regions-in-transition timeout period so we tried to reassign. The master aborted because region was in the PENDING_OPEN state when we went about assigning.
> {code}
> 2010-10-01 16:02:28,809 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: usertable,user1128734802,1285701924906.006696a9bf346f8593df66728e18e029. state=PENDING_OPEN, ts=1285948921051
> 2010-10-01 16:02:28,809 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_OPEN or OPENING for too long, reassigning region=usertable,user1128734802,1285701924906.006696a9bf346f8593df66728e18e029.
> 2010-10-01 16:02:28,811 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state trying to OFFLINE; usertable,user1128734802,1285701924906.006696a9bf346f8593df66728e18e029. state=PENDING_OPEN, ts=1285948921051
> java.lang.IllegalStateException
> at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:662)
> at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:632)
> at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:560)
> at org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor.chore(AssignmentManager.java:1102)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.