You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2024/09/05 02:55:00 UTC
[jira] [Commented] (HBASE-28815) Upgrade from 1.7.2 to 2.6.0 failed: HMaster aborted
[ https://issues.apache.org/jira/browse/HBASE-28815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879405#comment-17879405 ]
Duo Zhang commented on HBASE-28815:
-----------------------------------
You can try upgrade to 2.4.18 first and then to 2.6.0?
> Upgrade from 1.7.2 to 2.6.0 failed: HMaster aborted
> ---------------------------------------------------
>
> Key: HBASE-28815
> URL: https://issues.apache.org/jira/browse/HBASE-28815
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.6.0
> Reporter: Ke Han
> Priority: Major
>
> I am trying to migrate from 1.7.2 cluster to 2.6.0 (both are released versions). However, I observed that the hmaster crashed during the upgrade process.
> h1. Reproduce
> Step1: Start up 1.7.2 HBase cluster (1 HDFS, 1 HM, 1 RS).
> Step2: Stop the 1.7.2 HBase cluster.
> Step3: Upgrade to 2.6.0 HBase cluster.
> HMaster will crash with the following exception
> {code:java}
> 2024-09-04T16:04:47,004 WARN [PEWorker-2] procedure.InitMetaProcedure: Failed to init meta, suspend 1000secs
> java.io.IOException: Meta table is not partial, please sideline this meta directory or run HBCK to fix this meta table, e.g. rebuild the server hostname in ZNode for the meta region
> at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:944) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1766) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1444) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:77) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:2092) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) ~[hbase-common-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2119) ~[hbase-procedure-2.6.0.jar:2.6.0]
> 2024-09-04T16:04:47,005 INFO [PEWorker-2] procedure2.TimeoutExecutorThread: ADDED pid=1, state=WAITING_TIMEOUT:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta; timeout=1000, timestamp=1725465888005
> 2024-09-04T16:04:48,045 ERROR [PEWorker-1] procedure2.ProcedureExecutor: Root Procedure pid=1, state=FAILED:INIT_META_WRITE_FS_LAYOUT, exception=org.apache.hadoop.hbase.exceptions.TimeoutIOException via ProcedureExecutor:org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec; InitMetaProcedure table=hbase:meta does not support rollback but the execution failed and try to rollback, code bug?
> org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
> ... 3 more
> 2024-09-04T16:04:48,058 INFO [PEWorker-1] procedure2.ProcedureExecutor: Rolled back pid=1, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.exceptions.TimeoutIOException via ProcedureExecutor:org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec; InitMetaProcedure table=hbase:meta exec-time=1.4160 sec
> 2024-09-04T16:04:48,059 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: Failed to become active master
> java.io.IOException: Failed to initialize meta table
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1077) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2459) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.lambda$null$0(HMaster.java:590) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-common-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177) ~[hbase-common-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$1(HMaster.java:587) ~[hbase-server-2.6.0.jar:2.6.0]
> at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
> Caused by: org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
> 2024-09-04T16:04:48,060 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.quotas.MasterQuotasObserver]
> 2024-09-04T16:04:48,060 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hmaster,16000,1725465872458: Unhandled exception. Starting shutdown. *****
> java.io.IOException: Failed to initialize meta table
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1077) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2459) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.lambda$null$0(HMaster.java:590) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-common-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177) ~[hbase-common-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$1(HMaster.java:587) ~[hbase-server-2.6.0.jar:2.6.0]
> at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
> Caused by: org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
> Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
> at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
> 2024-09-04T16:04:48,065 INFO [master/hmaster:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'hmaster,16000,1725465872458' *****
> 2024-09-04T16:04:48,065 INFO [master/hmaster:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/hmaster:16000:becomeActiveMaster
> 2024-09-04T16:04:48,067 INFO [master/hmaster:16000] regionserver.HRegionServer: Stopping infoServer
> 2024-09-04T16:04:48,169 INFO [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.w.WebAppContext@78483404{master,/,null,STOPPED}{file:/hbase/hbase-2.6.0/hbase-webapps/master}
> 2024-09-04T16:04:48,180 INFO [master/hmaster:16000] server.AbstractConnector: Stopped ServerConnector@3a7ba368{HTTP/1.1, (http/1.1)}{0.0.0.0:16010}
> 2024-09-04T16:04:48,180 INFO [master/hmaster:16000] server.session: node0 Stopped scavenging
> 2024-09-04T16:04:48,182 INFO [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@b525e3a{static,/static,file:///hbase/hbase-2.6.0/hbase-webapps/static/,STOPPED}
> 2024-09-04T16:04:48,183 INFO [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@294e4c35{logs,/logs,file:///var/log/hbase/,STOPPED}
> 2024-09-04T16:04:48,188 INFO [master/hmaster:16000] regionserver.HRegionServer: aborting server hmaster,16000,1725465872458
> 2024-09-04T16:04:48,190 INFO [master/hmaster:16000] regionserver.HRegionServer: stopping server hmaster,16000,1725465872458; all regions closed.
> 2024-09-04T16:04:48,191 WARN [OldWALsCleaner-1] cleaner.LogCleaner: Interrupted while cleaning old WALs, will try to clean it next round. Exiting.
> 2024-09-04T16:04:48,191 WARN [OldWALsCleaner-0] cleaner.LogCleaner: Interrupted while cleaning old WALs, will try to clean it next round. Exiting.
> 2024-09-04T16:04:48,194 INFO [master/hmaster:16000] hbase.ChoreService: Chore service for: master/hmaster:16000 had [] on shutdown
> 2024-09-04T16:04:48,196 INFO [master/hmaster:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher
> 2024-09-04T16:04:48,196 INFO [master/hmaster:16000] procedure2.ProcedureExecutor: Stopping
> 2024-09-04T16:04:48,205 INFO [master/hmaster:16000] region.RegionProcedureStore: Stopping the Region Procedure Store, isAbort=true
> 2024-09-04T16:04:48,208 WARN [master/hmaster:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
> 2024-09-04T16:04:48,208 INFO [master/hmaster:16000] assignment.AssignmentManager: Stopping assignment manager
> 2024-09-04T16:04:48,209 INFO [master/hmaster:16000] region.MasterRegion: Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true
> 2024-09-04T16:04:48,369 INFO [master/hmaster:16000] regionserver.HRegion: Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682.
> 2024-09-04T16:04:48,373 ERROR [master/hmaster:16000] regionserver.HRegion: Memstore data size is 2190 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682.
> 2024-09-04T16:04:48,374 INFO [master/hmaster:16000] regionserver.HRegion: Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682.
> 2024-09-04T16:04:48,374 INFO [master/hmaster:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
> 2024-09-04T16:04:48,374 INFO [master:store-WAL-Roller] wal.AbstractWALRoller: LogRoller exiting.
> 2024-09-04T16:04:48,376 INFO [master/hmaster:16000] ipc.NettyRpcServer: Stopping server on /192.168.227.2:16000
> 2024-09-04T16:04:48,511 INFO [master/hmaster:16000] zookeeper.ZooKeeper: Session: 0x10009e5081e0000 closed
> 2024-09-04T16:04:48,511 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10009e5081e0000
> 2024-09-04T16:04:48,511 INFO [master/hmaster:16000] regionserver.HRegionServer: Exiting; stopping=hmaster,16000,1725465872458; zookeeper connection closed.
> 2024-09-04T16:04:48,512 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: HMaster Aborted
> at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:255) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:147) ~[hbase-server-2.6.0.jar:2.6.0]
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) ~[hadoop-common-2.10.2.jar:?]
> at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140) ~[hbase-server-2.6.0.jar:2.6.0] {code}
>
> Upgrade from 1.7.2 to 2.4.18 can still succeed.
> I am wondering whether this is a backward incompatibility or if there are any additional steps I missed to upgrade to 2.6.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)