[jira] [Commented] (HBASE-7101) HBase stuck in Region SPLIT

["Ted Yu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491179#comment-13491179 ] 

Ted Yu commented on HBASE-7101:
-------------------------------

The following code is from 0.94:
{code}
          Thread.sleep(100);
          // When this returns -1 it means the znode doesn't exist
          this.znodeVersion = tickleNodeSplit(server.getZooKeeper(),
            parent.getRegionInfo(), a.getRegionInfo(), b.getRegionInfo(),
            server.getServerName(), this.znodeVersion);
          spins++;
        } while (this.znodeVersion != -1 && !server.isStopped()
            && !services.isStopping());
{code}
In 0.92, condition for while loop is (this.znodeVersion != -1)
Meaning, we check whether the znode exists before writing new version.
                
> HBase stuck in Region SPLIT 
> ----------------------------
>
>                 Key: HBASE-7101
>                 URL: https://issues.apache.org/jira/browse/HBASE-7101
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.92.1
>            Reporter: Bing Jiang
>
> I found this issue from a zknode which has existed for a long time in the unassigned parent.And HMaster report warnning log increasingly.The loop log is at below. 
> WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 1a1c950ad45812d7b4b9b90ebf268468 not found on server sev0040,60020,1350378314041; failed processing
> WARN org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 1a1c950ad45812d7b4b9b90ebf268468 from server sev0040,60020,1350378314041 but it doesn't exist anymore, probably already processed its split
> WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 1a1c950ad45812d7b4b9b90ebf268468 not found on server gs-dpo-sev0040,60020,1350378314041; failed processing
> WARN org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 1a1c950ad45812d7b4b9b90ebf268468 from server sev0040,60020,1350378314041 but it doesn't exist anymore, probably already processed its split
> we use Hbase-0.92.1, and I trace back to the source code. HMaster AssignmentManager have already deleted the SPLIT_Region in its memory structure,but HRegionServer SplitTransaction has found the unassigned/parent-node existed in a transient state, precisely SplitTransaction executes tickleNodeSplit to update a new version a little later than  AssignmentManager deleting unassigned/parent-znode. After updating a version of the znode, it will intrigue the handleRegion operation again, however, AssignmentManager assert that the RegionState in Memory has been deleted, and transaction goes into a retry loop.
> In the SplitTransaction, transitionZKNode will retry tickleNodeSplit after sleeping 100ms. In my opinion, if the time is much longger than 100ms, all the operation from AssignmentManagement will finish off completely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira