[jira] [Created] (HBASE-8545) Meta stuck in transition when it is assigned to a just restarted dead region sever

["Jimmy Xiang (JIRA)" <ji...@apache.org>]

Jimmy Xiang created HBASE-8545:
----------------------------------

             Summary: Meta stuck in transition when it is assigned to a just restarted dead region sever 
                 Key: HBASE-8545
                 URL: https://issues.apache.org/jira/browse/HBASE-8545
             Project: HBase
          Issue Type: Bug
          Components: Region Assignment
            Reporter: Jimmy Xiang
            Assignee: Jimmy Xiang


Support the meta region server is down, and the SSH tries to re-assign it.  This could happen:

1. AM plans to assign meta to a region server (R_old);
2. Now R_old is dead, the new region server (R_new) starts up on the same host, port, but gets a different start code;
3. AM sends the open region request to R_new and the Meta is opened on it;
4. AM gets ZK event, but it is from a different region server instance (R_new), not the expected one (R_old), so it sends a close region request to R_new;
5. Now, the meta is stuck in transition and won't be assigned.

This won't happen to a user region since the SSH for R_old will find out the user region stuck in transition and re-assign it.  For meta, it is a little different.  AM checks if a dead region server carries the meta based on the ZK info, which is changed to the new region server R_new at step 3 by the open region handler.

The fix I was thinking about is:
1. In checking is a region server carries a region, uses the region transition information if it exists (which is the source of truth, to master), if not, checks the ZK data as before;
2. In open region handler, when transition assign zk node from offline to opening, make sure the current region server is the expected one (ZK#transitionNode, existing code doesn't check the target server name).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira