[jira] [Created] (HBASE-16276) Inconsistency in Meta on cluster initialization and inability to assign regions on cluster restarts

["Joseph (JIRA)" <ji...@apache.org>]

Joseph created HBASE-16276:
------------------------------

             Summary: Inconsistency in Meta on cluster initialization and inability to assign regions on cluster restarts 
                 Key: HBASE-16276
                 URL: https://issues.apache.org/jira/browse/HBASE-16276
             Project: HBase
          Issue Type: Bug
            Reporter: Joseph
            Priority: Critical


I have recently been experimenting with a cluster running on the latest Master commit: 519f87f 
I initialize a cluster of 11 RegionServers and 3 HMaster's and then create a giant pre-split table using HBase Pe (~500 regions). I then kill the entire cluster while the HBase Pe test is still writing to the cluster. When I restart the cluster, a single RS will somehow always fail to open up any regions. There will also be a large number of regions stuck in transition with either Pending_Close or Failed_Close on a variety of RS's in the cluster.

Running Hbase hbck when I first initialized the cluster produces:

***********************************************************************

HBaseFsck command line options: 
Version: 2.0.0-fb10-SNAPSHOT
Number of live region servers: 12
Number of dead region servers: 0
Master: hbasectrl377.ash2.facebook.com,16000,1469225695984
Number of backup masters: 2
Average load: 0.16666666666666666
Number of requests: 0
Number of regions: 2
Number of regions in transition: 0
ERROR: hbase:meta, replicaId 0 is found on more than one region.
ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting...

Summary:
2 inconsistencies detected.
Status: INCONSISTENT

***********************************************************************

Finally in the restarted cluster. The Meta region is assigned a region, which looks very different from normal region names.
"hbase:meta,,1"
as opposed to
"hbase:meta,,1.1588230740"





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)