You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2013/01/31 01:45:13 UTC
[jira] [Commented] (HBASE-7724) [0.94] Just OPENED regions are
ignored by ServerShutdownHandler and go unassigned for ever
[ https://issues.apache.org/jira/browse/HBASE-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567173#comment-13567173 ]
stack commented on HBASE-7724:
------------------------------
{code}
2013-01-30 02:37:25,997 DEBUG org.apache.hadoop.hbase.master.ServerManager: STARTUP: Server XX.XX.18.40,10304,1359513445136 came back up, removed it from the dead servers list
2013-01-30 02:40:46,573 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=OBFUSCATED_TABLE,OBFUSCATED_STARTKEY,1344723216908.55e9cb551edeea0b52bb91af7c2de199., src=sv4r20s16,10304,1357578057026, dest=XX.XX.18.40,10304,1359513445136
...
2013-01-30 02:41:04,995 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=XX.XX.18.40,10304,1359513445136, region=55e9cb551edeea0b52bb91af7c2de199
2013-01-30 02:41:04,995 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for OBFUSCATED_TABLE,OBFUSCATED_STARTKEY,1344723216908.55e9cb551edeea0b52bb91af7c2de199. from XX.XX.18.40,10304,1359513445136; deleting unassigned node
2013-01-30 02:41:05,046 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:10302-0xd3b032ca78e76dc Retrieved 112 byte(s) of data from znode /prodjobs/unassigned/55e9cb551edeea0b52bb91af7c2de199; data=region=OBFUSCATED_TABLE,\x09\xC9\xA1\xB2 u\xF1\x82yfd\xCC"\xBCG},1344723216908.55e9cb551edeea0b52bb91af7c2de199., origin=XX.XX.18.40,10304,1359513445136, state=RS_ZK_REGION_OPENED
..
2013-01-30 02:41:19,917 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Skip assigning region OBFUSCATED_TABLE,OBFUSCATED_STARTKEY,1344723216908.55e9cb551edeea0b52bb91af7c2de199. state=OPEN, ts=1359513674715, server=XX.XX.18.40,10304,1359513445136
{code}
Here is what is on the regionserver
{code}
2013-01-30 02:40:54,429 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened OBFUSCATED_TABLE,OBFUSCATED_STARTKEY,1344723216908.55e9cb551edeea0b52bb91af7c2de199. on server:XX.XX.18.40,10304,1359513445136
Wed Jan 30 02:41:00 UTC 2013 Terminating regionserver
2013-01-30 02:41:01,016 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region OBFUSCATED_TABLE,OBFUSCATED_STARTKEY,1344723216908.55e9cb551edeea0b52bb91af7c2de199.
{code}
> [0.94] Just OPENED regions are ignored by ServerShutdownHandler and go unassigned for ever
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-7724
> URL: https://issues.apache.org/jira/browse/HBASE-7724
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Reporter: stack
>
> Visiting a user today, I came across following interesting case (0.94.2 HBase).
> A server was added to cluster. It was assigned regions by the balancer. A bunch opened on the regionserver and just after the open, the regionserver was manually shutdown. There was a lag processing the zk region open events in the master (because clean shutdown, there was no zk activity when the regions were closed on the shutdown regionserver). Processing the server shutdown, we do this for a good few of the regions that had just been opened on the regionserver:
> 2013-01-30 02:41:19,917 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Skip assigning region OBFUSCATED_TABLE,OBFUSCATED_STARTROW,1344723216908.55e9cb551edeea0b52bb91af7c2de199. state=OPEN, ts=1359513674715, server=XX.XX.18.40,10304,1359513445136
> Seems like outright bug that'd we'd skip a region that is in transition that is in the OPEN state.
> More detail to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira