You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Yongle Zhang (Jira)" <ji...@apache.org> on 2020/11/09 23:58:00 UTC
[jira] [Created] (HBASE-25259) After upgrading HBase from 2.1.1 to
2.2.0, table cannot be read and HMaster crashes
Yongle Zhang created HBASE-25259:
------------------------------------
Summary: After upgrading HBase from 2.1.1 to 2.2.0, table cannot be read and HMaster crashes
Key: HBASE-25259
URL: https://issues.apache.org/jira/browse/HBASE-25259
Project: HBase
Issue Type: Bug
Affects Versions: 2.1.1, 2.2.0
Reporter: Yongle Zhang
When we upgraded HBASE cluster from 2.1.1 to 2.2.0, we failed to read data by using HBase shell command, and later HMaster crashes.
Commandline error message:
{code:java}
HBase ShellUse "help" to get list of supported commands.Use "exit" to quit this interactive shell.For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion 2.2.0, r3ec693241f2edb0cf78e71739874f2323a86bf1e, Tue Nov 3 02:12:43 UTC 2020Took 0.0043 secondshbase(main):001:0> listTABLETestTable1 row(s)Took 0.5663 seconds=> ["TestTable"]hbase(main):002:0> scan 'TestTable'ROW COLUMN+CELL
ERROR: Unknown table TestTable!
For usage try 'help "scan"'
Took 0.1510 secondshbase(main):003:0>
{code}
Log file:
{code:java}
2020-11-07 04:03:57,346 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: []2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master bff37223dfc6,16000,1604721496292: Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,348 INFO [master/bff37223dfc6:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'bff37223dfc6,16000,1604721496292' *****2020-11-07
stopping=bff37223dfc6,16000,1604721496292; zookeeper connection closed.2020-11-07 04:03:57,612 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1014e7bdcf800002020-11-07 04:03:57,620 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: HMaster Abortedat org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
{code}
Steps to reproduce:
# ** Start up a cluster of version 2.1.1 with 3 nodes
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
# Stop the cluster by:
## run “bash stop-hbase.sh” to shutdown the master.
## Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml
## Start the master again and wait until the Master quits
# Upgrade the node to 2.2.0
# After upgrading, try to read the data TestTable, the error message shows. Then, after a while, Master crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)