[jira] [Created] (HDDS-5546) OM Service ID change causes OM startup failure

["Bharat Viswanadham (Jira)" <ji...@apache.org>]

Bharat Viswanadham created HDDS-5546:
----------------------------------------

             Summary: OM Service ID change causes OM startup failure
                 Key: HDDS-5546
                 URL: https://issues.apache.org/jira/browse/HDDS-5546
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Bharat Viswanadham
            Assignee: Bharat Viswanadham


In OM HA, raftGroupID is generated from service ID.
So, if there is a change in OM Service ID OM startup fails with below error


{code:java}
2021-08-05 12:20:03,043 ERROR org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.io.IOException: java.lang.IllegalStateException: ILLEGAL TRANSITION: In OzoneManagerStateMachine:om1:group-8A65FD498CB6, RUNNING -> STARTING
        at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
        at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
        at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:71)
        at org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:354)
        at org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:371)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.start(OzoneManagerRatisServer.java:390)
        at org.apache.hadoop.ozone.om.OzoneManager.start(OzoneManager.java:1109)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:126)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:79)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:67)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:38)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
        at picocli.CommandLine.access$1100(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2152)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2530)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2465)
        at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:96)
        at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87)
        at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:51)
Caused by: java.lang.IllegalStateException: ILLEGAL TRANSITION: In OzoneManagerStateMachine:om1:group-8A65FD498CB6, RUNNING -> STARTING
        at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
        at org.apache.ratis.util.LifeCycle$State.validate(LifeCycle.java:121)
        at org.apache.ratis.util.LifeCycle.transition(LifeCycle.java:164)
        at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:268)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.initialize(OzoneManagerStateMachine.java:127)
        at org.apache.ratis.server.impl.ServerState.<init>(ServerState.java:120)
        at org.apache.ratis.server.impl.RaftServerImpl.<init>(RaftServerImpl.java:193)
        at org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$4(RaftServerProxy.java:266)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}


The reason is now a new ratis group directory is created and StateMachine instance is shared between them. The error is confusing to end users as it is not clear that it is due to change in OM serviceId this caused failure.

This Jira is to add some safeguard code and give clear message to know om startup failure. I will raise another jira to not to use om service id in ratis group ID.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org