[jira] [Created] (HBASE-25260) upgrading hbase from 2.0.6 to 2.1.1, HMaster failed to become active because it cannot find hbase:namespace table

["Yongle Zhang (Jira)" <ji...@apache.org>]

Yongle Zhang created HBASE-25260:
------------------------------------

             Summary: upgrading hbase from 2.0.6 to 2.1.1, HMaster failed to become active because it cannot find hbase:namespace table
                 Key: HBASE-25260
                 URL: https://issues.apache.org/jira/browse/HBASE-25260
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.0.6, 2.1.1
            Reporter: Yongle Zhang
         Attachments: hmaster.log

When we upgraded HBASE cluster from 2.0.6 to 2.1.1, the HMaster on upgraded node failed to start.

Some stack trace in the error log:
{code:java}
2020-11-06 02:01:26,420 WARN  [PEWorker-12] assignment.RegionTransitionProcedure: Failed transition, suspend 1secs pid=12, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=TestTable, region=37d62d2c1934da269a592e0e5cbca82a; rit=OFFLINE, location=null; waiting on rectified condition fixed by other Procedure or operator intervention
org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: TestTable
  at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:215)
  at org.apache.hadoop.hbase.master.assignment.AssignProcedure.assign(AssignProcedure.java:194)
  at org.apache.hadoop.hbase.master.assignment.AssignProcedure.startTransition(AssignProcedure.java:205)
  at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:355)
  at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:957)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1835)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1595)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1200(ProcedureExecutor.java:80)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2140)

{code}
Seems it's caused by not able to find hbase:namespace table after upgrade: 
{code:java}
2020-11-06 02:01:26,791 ERROR [master/399fd6ca0c6d:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: []
2020-11-06 02:01:26,791 ERROR [master/399fd6ca0c6d:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 399fd6ca0c6d,16000,1604628075265: Unhandled exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
  at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
  at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
  at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1253)
  at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1031)
  at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2254)
  at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:583)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace
  at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:864)
  at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:759)
  at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131)
  at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:745)
  at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131)
  at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:716)
  at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131)
  at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:594)
  at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.getRegionLocation(ConnectionUtils.java:131)
  at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:72)
  at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:223)
  at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105)
  at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)
  at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)
  at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:142)
  at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:279)
  at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
  at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)
  at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
  at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1251)
  ... 4 more
{code}
Attached the error log file. 

[^hmaster.log]

 

Steps to reproduce: 
 # Start up a cluster of version 2.0.6 with 3 nodes
 # Use hbase pe to write data. 

{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
 # Stop the cluster:  
 ## using the graceful_stop.sh to stop all regionservers.
 ## Then run stop-hbase.sh
 # Upgrade the node to 2.1.1

5. After upgrading HMaster failed to start.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)