You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Arun Sarin (Jira)" <ji...@apache.org> on 2026/04/02 09:22:00 UTC
[jira] [Updated] (HDDS-14964) Write failures with ContainerNotOpenException during large file (100GB) concurrent parallel writes at high cluster utilization (~90%)
[ https://issues.apache.org/jira/browse/HDDS-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun Sarin updated HDDS-14964:
------------------------------
Issue Type: Bug (was: Improvement)
> Write failures with ContainerNotOpenException during large file (100GB) concurrent parallel writes at high cluster utilization (~90%)
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-14964
> URL: https://issues.apache.org/jira/browse/HDDS-14964
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Arun Sarin
> Priority: Major
>
> During a large-scale deletion test, the cluster was being filled with 100GB files nested across various directory structures. Writes were progressing successfully until ~90% disk utilization was reached on approximately 80% of the DataNodes. At that point, all large file writes started failing with {{{}ContainerNotOpenException{}}}.
> Key observation: Small file writes on the same cluster succeeded without issue during this same time window. Only concurrent parallel writes of large files (100GB) were failing. This suggests the issue is not that there are zero writable containers on the cluster, but rather that the client — while in the middle of writing a large file — encounters a container that has transitioned to {{CLOSED}} state and is unable to recover or allocate a new one successfully in that context.
> *Root cause hypothesis:* When writing a 100GB file, many chunks are sent sequentially to the same container. At high cluster utilization, a container can transition to {{CLOSED}} state mid-write (e.g., due to being full or the SCM closing it). The client hits {{ContainerNotOpenException}} on a {{WriteChunk}} call. While retry/failover logic exists, under high disk pressure the client is unable to get a new OPEN container allocated in time, causing the write to fail entirely.
> Error Stack :
> {code:java}
> java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 20677 in CLOSED state
> at org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
> at org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
> at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
> at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:105)
> at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:66)
> at org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:147)
> at org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:351)
> at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequestWithRetry$5(OrderedAsync.java:210)
> at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:325)
> at java.base/java.util.Optional.ifPresent(Optional.java:178)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:381)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:325)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
> at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
> at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 20677 in CLOSED state
> at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
> at org.apache.ratis.server.impl.RaftServerImpl.writeAsyncImpl(RaftServerImpl.java:987)
> at org.apache.ratis.server.impl.RaftServerImpl.writeAsync(RaftServerImpl.java:960)
> at org.apache.ratis.server.impl.RaftServerImpl.replyFuture(RaftServerImpl.java:953)
> at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:930)
> at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$11(RaftServerImpl.java:919)
> at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
> at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$12(RaftServerImpl.java:919)
> at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
> ... 3 more
> Caused by: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: Container 20677 in CLOSED state
> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:265)
> at org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:455)
> at org.apache.ratis.client.impl.ClientProtoUtils.toStateMachineException(ClientProtoUtils.java:441)
> at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:408)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:313)
> at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:308)
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:551)
> at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
> at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:648)
> at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> ... 3 more
> 26/03/02 23:09:43 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> traceID: ""
> containerID: 20677
> datanodeUuid: "c15cc65e-f4eb-43f0-b27c-6b6a3fbc5a86"
> writeChunk {
> blockID {
> containerID: 20677
> localID: 117883640217932227
> blockCommitSequenceId: 686720
> replicaIndex: 0
> }
> chunkData {
> chunkName: "117883640217932227_chunk_53"
> offset: 218103808
> len: 4194304
> checksumData {
> type: CRC32
> bytesPerChecksum: 1048576
> checksums: "\216\301\375\321"
> checksums: "\300\2022\n"
> checksums: "\334\002L0"
> checksums: "\335m=\322"
> }
> }
> } {code}
> *Steps to Reproduce:*
> # Set up an Apache Ozone cluster with sufficient DataNodes (e.g., 10+).
> # Write 100GB files concurrently across various directory structures using Freon (or similar tool) targeting petabyte-scale data.
> # Continue writing until approximately 80–90% disk utilization is reached on the majority of DataNodes.
> # Observe that large file writes begin failing with {{{}ContainerNotOpenException{}}}, while small file writes on the same cluster continue to succeed.
> *Expected Behavior:*
> Large file writes should either:
> * Successfully obtain a new OPEN container when the current container transitions to CLOSED mid-write and continue writing, OR
> * Fail early with a clear, user-facing error (e.g., "Insufficient cluster space") before the write is initiated, rather than failing mid-stream after chunks have already been sent.
> Actual Behavior:
> Large file (100GB) writes fail mid-stream with:
> java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
> from Server 098be32c-a26e-4fe9-a491-c7d17b9bb04b@group-D9554F414C68: Container 20677 in CLOSED state
> at org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:373)
> at org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:175)
> ...
> Caused by: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException: Container 20677 in CLOSED state
> at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.validateContainerCommand(HddsDispatcher.java:581)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.startTransaction(ContainerStateMachine.java:488)
> ...
>
> Failed {{WriteChunk}} request details:
> * Container ID: 20677
> * Block local ID: 117883640217932227
> * Chunk: {{{}117883640217932227_chunk_53{}}}, offset: 218103808, len: 4194304
> *Proposed Improvement:*
> Similar to how quota validation is performed before a key {{PUT}} (checking namespace/space quota against available capacity), a pre-write cluster-level space check should be introduced before initiating a large file write. Specifically:
> * Before allocating block pipelines for a write, check whether the cluster has sufficient OPEN/allocable containers to accommodate the expected write size.
> * If the cluster is approaching full capacity and cannot allocate the required containers, fail fast with a clear error (e.g., {{ClusterStorageFullException}} or similar) rather than allowing the write to proceed and fail mid-stream with a confusing {{{}ContainerNotOpenException{}}}.
> This would provide better UX, avoid partial writes/orphaned blocks, and align with the existing quota-check pattern already present at the OM layer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org