You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2025/12/01 21:35:00 UTC
[jira] [Commented] (HDDS-13842) Follower SCM does not comes out of safemode
[ https://issues.apache.org/jira/browse/HDDS-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18041979#comment-18041979 ]
Ethan Rose commented on HDDS-13842:
-----------------------------------
If this incorporates HDDS-13980 and HDDS-13981 then it is expected to contain a fix for those issues as well and they can be resolved. Otherwise, a different link type should be used.
> Follower SCM does not comes out of safemode
> -------------------------------------------
>
> Key: HDDS-13842
> URL: https://issues.apache.org/jira/browse/HDDS-13842
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Affects Versions: 2.1.0
> Reporter: Sumit Agrawal
> Assignee: Sumit Agrawal
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.2.0
>
>
> Follower SCM never comes out of safe mode, as RPC server for DN HB is not started at SCM. Its started if certain action like leader change happens, Or leader is doing some update to SCM.
>
> scm3: follower on startup
>
> {code:java}
> 2025-10-15 17:39:57,265 INFO [main]-org.apache.hadoop.hdds.scm.node.SCMNodeManager: Entering startup safe mode.
> <-- SCM RPC server is not started and hence do not receive DN HB.
> 2025-10-15 17:41:10,344 INFO ScmDatanodeProtocol RPC server for DataNodes
> <-- RPC started as above log
> 2025-10-15 17:42:29,287 INFO [node3-EventQueue-ContainerRegistrationReportForRatisContainerSafeModeRule]-org.apache.hadoop.hdds.scm.safemode.SCMSafeModeManager: SCM exiting safe mode.
> {code}
>
>
> Another observation where no action performed for 10 min:
>
> {code:java}
> 2025-10-25 19:19:34,473 INFO .. Entering startup safe mode.
> <-- 11 min delay as test case waiting for follower to exit safemode and no action
> 2025-10-25 19:30:47,921 INFO .. ScmDatanodeProtocol RPC server for DataNodes
> {code}
>
>
> This is as, its started on notifyTermIndexUpdated() call from ratis. below is call flow.
> {code:java}
> 2025-10-17 14:10:46,937 ERROR [b6e60709-ec61-4360-8fb3-65b2317949c0@group-29860CDEEB45-StateMachineUpdater]-org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer: heyho
> java.lang.Exception
> at org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.start(SCMDatanodeProtocolServer.java:199)
> at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyTermIndexUpdated(SCMStateMachine.java:364)
> at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1848)
> at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:252)
> at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:193)
> at java.lang.Thread.run(Thread.java:748) {code}
>
> As solution:
> 1) DatanodeProtocolServer can be started immediately when scm starts so that DN register/re-register can happen immediately
> 2) revert Ratis log metadata - ensure the termUpdate event happens for metadata changes also
> 2) StateMachineReadyRule triggered by StateMachine using refreshAndvalidate() all rules, ensure all rules are re-checked, and exit safemode status after start machine has flushed / applied all raft log on startup to avoid in-progress raft log transaction issue as referred by HDDS-5263.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org