You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org> on 2011/09/02 18:57:12 UTC
[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make
it less racy
[ https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096126#comment-13096126 ]
ramkrishna.s.vasudevan commented on HBASE-4015:
-----------------------------------------------
The following tests were performed
The cluster has 3 nodes.
Created some 4000+ regions. And i followed the startegy like
Initially create regions
Balancer will balance the regions across all the three.
Using a script kill the RS in these 3 nodes at fixed time.(at different time intervals).
Then start it after sometime.
Sometimes once all the 3 RS are up invoke the balancer command manually.
While balancing kill one of the RS which was the destination during balancer calculation.
Again bring it up.
The number of regions remained constant and every time the Timeout detected it was able to overcome it with no issues at all.
I also calculated some metrics
There were instance were 144, 249,1334, 30 regions were timed out.
For any timeout region to be monitored and completed it took at the max 1.5 sec minimum being 0.35 secs.
The following logs shows the scenario where the timeout monitor tries to assign a node to OFFLINE but by the time the node is changed to OPENED by RS.
{code}
2011-09-02 18:15:55,621 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_OPEN for too long, reassigning region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.
2011-09-02 18:15:55,709 INFO org.mortbay.log: Timeout detected for the region testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88. at Fri Sep 02 18:15:55 IST 2011
2011-09-02 18:15:55,942 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x13229c29f830010 Creating (or updating) unassigned node for b7a45cf31651d5cbba682ae24b83bf88 with OFFLINE state
2011-09-02 18:15:55,967 INFO org.apache.hadoop.hbase.zookeeper.ZKAssign: Current data in znode .region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88., origin=HOST-10-18-52-198,60020,1314967317396, state=RS_ZK_REGION_OPENED
2011-09-02 18:15:55,996 WARN org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force node into OFFLINE state before completing assignment but failed to do so for testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88. state=PENDING_OPEN, ts=1314967555942, server=null
2011-09-02 18:15:55,999 INFO org.mortbay.log: Time taken to complete the action on timeout for the reigon testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.291
2011-09-02 18:16:24,115 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=HOST-10-18-52-198,60020,1314967317396, region=b7a45cf31651d5cbba682ae24b83bf88, which is more than 15 seconds late
2011-09-02 18:16:24,122 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x13229c29f830010 Successfully deleted unassigned node for region b7a45cf31651d5cbba682ae24b83bf88 in expected state RS_ZK_REGION_OPENED
{code}
> Refactor the TimeoutMonitor to make it less racy
> ------------------------------------------------
>
> Key: HBASE-4015
> URL: https://issues.apache.org/jira/browse/HBASE-4015
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 0.90.3
> Reporter: Jean-Daniel Cryans
> Assignee: ramkrishna.s.vasudevan
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBASE-4015_1_trunk.patch, HBASE-4015_2_trunk.patch, HBASE-4015_reprepared_trunk_2.patch, Timeoutmonitor with state diagrams.pdf
>
>
> The current implementation of the TimeoutMonitor acts like a race condition generator, mostly making things worse rather than better. It does it's own thing for a while without caring for what's happening in the rest of the master.
> The first thing that needs to happen is that the regions should not be processed in one big batch, because that sometimes can take minutes to process (meanwhile a region that timed out opening might have opened, then what happens is it will be reassigned by the TimeoutMonitor generating the never ending PENDING_OPEN situation).
> Those operations should also be done more atomically, although I'm not sure how to do it in a scalable way in this case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira