[jira] [Commented] (HBASE-21440) Assign procedure on the crashed server is not properly interrupted

["Allan Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676234#comment-16676234 ] 

Allan Yang commented on HBASE-21440:
------------------------------------

Yes, I think it is possible. But it should be a very rare case. Since if an AssignProcedure is in REGION_TRANSITION_FINISH state, it should have finished long time ago before the corresponding SCP begin to handle RIT. But, anyway, it can happen, e.g. the meta table is also on the crashed server, so the AssignProcedure is stucking there retrying to update meta, while sleeping, the SCP called remoteCallFailed on this AssignProcedure.
I think we can do this. if the procedure found in handleRIT is a assign procedure, and it is in REGION_TRANSITION_FINISH , we should not remove it from regions to assign list.
We can count on AssignProcedure's state, since, if it do not enter REGION_TRANSITION_FINISH state when handleRIT, it won't have a chance later(the server is already dead).

> Assign procedure on the crashed server is not properly interrupted
> ------------------------------------------------------------------
>
>                 Key: HBASE-21440
>                 URL: https://issues.apache.org/jira/browse/HBASE-21440
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ankit Singhal
>            Assignee: Ankit Singhal
>            Priority: Major
>             Fix For: 2.0.2
>
>
> When the server crashes, it's SCP checks if there is already a procedure assigning the region on this crashed server. If we found one, SCP will just interrupt the already running AssignProcedure by calling remoteCallFailed which internally just changes the region node state to OFFLINE and send the procedure back with transition queue state for assignment with a new plan. 
> But, due to the race condition between the calling of the remoteCallFailed and current state of the already running assign procedure(REGION_TRANSITION_FINISH: where the region is already opened), it is possible that assign procedure goes ahead in updating the regionStateNode to OPEN on a crashed server. 
> As SCP had already skipped this region for assignment as it was relying on existing assign procedure to do the right thing, this whole confusion leads region to a not accessible state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)