You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2026/09/12 02:32:00 UTC
[jira] [Assigned] (HDDS-16028) SCM follower suggests itself as leader can cause OM to be stuck
[ https://issues.apache.org/jira/browse/HDDS-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Andika reassigned HDDS-16028:
----------------------------------
Assignee: Ivan Andika (was: ChenXi)
> SCM follower suggests itself as leader can cause OM to be stuck
> ---------------------------------------------------------------
>
> Key: HDDS-16028
> URL: https://issues.apache.org/jira/browse/HDDS-16028
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Critical
>
> We have another major incident partly because of SCM follower kept suggesting itself as a leader
> {code:java}
> scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
> scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
> scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
> {code:java}
> com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException): Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested leader is Server:scm17:9863.
> at org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
> at org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
> at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
> at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
> at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
> at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
> at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
> , while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863. Trying to failover after sleeping for 2000ms. {code}
> Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed it, see below
> {code:java}
> NotLeaderException generateNotLeaderException() {
> if (!lifeCycle.getCurrentState().isRunning()) {
> return new NotLeaderException(getMemberId(), null, null);
> }
> RaftPeerId leaderId = state.getLeaderId();
> if (leaderId == null || leaderId.equals(getId())) {
> // No idea about who is the current leader. Or the peer is the current
> // leader, but it is about to step down. set the suggested leader as null.
> leaderId = null;
> }
> final RaftConfigurationImpl conf = getRaftConf();
> Collection<RaftPeer> peers = conf.getAllPeers();
> return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
> } {code}
> But because Ozone (annoyingly) reinvented a (buggy) wheel of NotLeaderException generation, a non-leader SCM NotLeaderException with itself as the suggested leader
> {code:java}
> public NotLeaderException triggerNotLeaderException() {
> ByteString leaderId =
> division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
> .getId().getId();
> RaftPeer suggestedLeader = leaderId.isEmpty() ?
> null :
> division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
> return new NotLeaderException(division.getMemberId(),
> suggestedLeader,
> division.getGroup().getPeers());
> } {code}
> We should use the correct Ratis generatedNotLeaderException.
> This also apply to OM OzoneManagerRatisServer#newOMNotLeaderException, but thankfully OM suggested leadership mechanism is broken anyway so it will not be triggered (HDDS-14769).
> I will revisit all the Ozone Ratis usage and check if there are some similar issues. Most Ozone usage of Ratis need to be moved to Ratis. We will create new Ratis API to access it if necessary. If this causes Ozone to wait Ratis release every time, so be it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org