You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Krishna Kumar Asawa (Jira)" <ji...@apache.org> on 2026/06/10 09:42:04 UTC
[jira] [Assigned] (HDDS-14845) SCMNodeStat cannot be initialized when a negative scmUsed value has passed
[ https://issues.apache.org/jira/browse/HDDS-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Kumar Asawa reassigned HDDS-14845:
------------------------------------------
Assignee: Sreeja
> SCMNodeStat cannot be initialized when a negative scmUsed value has passed
> --------------------------------------------------------------------------
>
> Key: HDDS-14845
> URL: https://issues.apache.org/jira/browse/HDDS-14845
> Project: Apache Ozone
> Issue Type: Bug
> Environment: Apache Ozone 2.1.0 with HA-enabled, long-running datanodes with V2 & V3 containers, some corrupted containers.
> Reporter: Kohei Sugihara
> Assignee: Sreeja
> Priority: Major
>
> h2. Overview
> After upgrading to 2.1.0, we experienced slow file upload issue. Digging down this issue, we finally confirmed SCMNodeStat object could not be initialized due to a negative value report from datanodes. I think this is a small case issue because we've seen this issue on the long running datanodes since Apache Ozone 1.2.0, so most of recently configured instances do not affect this issue, but we share our workarounds and investigation.
> h2. Configuration
> * Apache Ozone 2.1.0 (or, with DUOptimized; it's default value since 2.1.0 by https://issues.apache.org/jira/browse/HDDS-12924)
> * Use the capacity-aware pipeline choosing policy
> * Datanode might have old containers (e.g. V2 schema) or corrupted containers
> h2. Workaround 1:
> We've seen the following stack trace about SCMNodeStat and this makes some retries during upload sequence.
> {code:java}
> 2026-03-02 13:49:06,818 | ERROR | SCMAudit | user=om/HOST@REALM | ip=IP_ADDR | op=ALLOCATE_BLOCK {"replication":"RATIS/THREE","owner":"omservice","size":"268435456","num":"1","client":""} | ret=FAILURE | java.lang.IllegalArgumentException: used space cannot be negative.
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
> at org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:46) {code}
> We configured our cluster with the capacity-aware pipeline choosing policy, so by switching to the random pipeline choosing policy resolved slow upload issue temporary.
> {code:java}
> <property>
> <name>hdds.scm.pipeline.choose.policy.impl</name>
> <value>org.apache.hadoop.hdds.scm.pipeline.choose.algorithms.CapacityPipelineChoosePolicy</value>
> </property> {code}
> h2. Workaround 2:
> We also examined the reason why a negative value was passed to SCMNodeStat. We've found the following logs on the datanode.
> {code:java}
> 2026-03-02 18:03:35,455 [c3e35eb9-a8be-463b-a22f-58b6e6e0767d-DataNodeDiskCheckerThread-60] WARN org.apache.hadoop.ozone.container.common.volume.HddsVolume: Volume /data/01/ozone
> /hdds reserved usages is higher than actual allocated reserved space.
> 2026-03-02 18:03:36,176 [DiskUsage-/data/00/ozone-
> ] INFO org.apache.hadoop.hdds.fs.DUOptimized: Disk metaPath du usages 36864, container data usages -877788034370
> 2026-03-02 18:03:36,178 [DiskUsage-/data/07/ozone-
> ] INFO org.apache.hadoop.hdds.fs.DUOptimized: Disk metaPath du usages 36864, container data usages -1388241327952
> 2026-03-02 18:03:36,196 [DiskUsage-/data/05/ozone-
> ... {code}
> We switched to the previous DU implementation to avoid this as our workaround. Our cluster has some containers in old schema versions and also including some corrupted containers, so it has possibly returned wrong value.
> {code:java}
> <property>
> <name>hdds.datanode.du.factory.classname</name>
> <value>org.apache.hadoop.hdds.fs.DUFactory</value>
> </property> {code}
> h2. Impact
> In addition to the capacity-aware pipeline choosing policy, the negative value in SCMNodeStat will occurs the following errors:
> h3. 1: {{ozone admin datanode usageinfo}} command fails without kind error message
> While debugging this, we also noticed {{ozone admin datanode}} command was stuck and fails. Raise their loglevel to DEBUG, we saw the following retries:
> {code:java}
> % ozone --loglevel DEBUG admin datanode usageinfo --json --hostname DN_HOST
> ...
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3198)
> , while invoking $Proxy20.submitRequest over nodeId=scm-2,nodeAddress=HOST after 6 failover attempts. Trying to failover after sleeping for 2000ms. Current retry count: 6.
> 2026-03-02 15:31:28,709 [main] DEBUG proxy.SCMContainerLocationFailoverProxyProvider: Failing over to next proxy. scm-2
> 2026-03-02 15:31:28,710 [IPC Parameter Sending Thread for HOST] DEBUG ipc.Client: IPC Client (1324262781) connection to HOST from USER@REALMIPC Client (1324262781) connection to HOST from USER@REALM sending #0 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest
> 2026-03-02 15:31:28,712 [IPC Client (1324262781) connection to HOST from USER] DEBUG ipc.Client: IPC Client (1324262781) connection to HOST from USER@REALM got value #0
> 2026-03-02 15:31:28,713 [main] DEBUG proxy.SCMContainerLocationFailoverProxyProvider: RetryProxy: SCM Server scm-2: RemoteException: used space cannot be negative.
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
> at org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:46)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1064)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.getUsageInfo(SCMNodeManager.java:1021)
> at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getUsageInfoFromDatanodeDetails(SCMClientProtocolServer.java:1359)
> at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getDatanodeUsageInfo(SCMClientProtocolServer.java:1338)
> at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getDatanodeUsageInfo(StorageContainerLocationProtocolServerSideTranslatorPB.java:1296)
> at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:673)
> at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
> at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:236)
> at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484)
> at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595)
> at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1228)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1246)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1169)
> at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3198)
> ...{code}
> When HA is enabled, the Ozone client asks to other SCM node and then receive we're not leader exception. We've seen the infinity error retries on the client side. The above logs are just excerpt from the long retries.
> h3. 2: Container Balancer fails unexpectedly
> Container Balancer uses SCMNodeStat object, so it fails. This is an error log excerpt from SCM log file.
> {code:java}
> 2026-03-02 17:06:10,168 [scm-1-ContainerBalancerTask-2] ERROR org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask: Container Balancer is stopped abnormally,
> java.lang.IllegalArgumentException: used space cannot be negative.
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
> at org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.<init>(SCMNodeStat.java:46)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.getNodeStatInternal(SCMNodeManager.java:1064)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.getUsageInfo(SCMNodeManager.java:1021)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.getMostOrLeastUsedDatanodes(SCMNodeManager.java:998)
> at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask.initializeIteration(ContainerBalancerTask.java:462)
> at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask.balance(ContainerBalancerTask.java:260)
> at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask.run(ContainerBalancerTask.java:190)
> at java.base/java.lang.Thread.run(Thread.java:840) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org