You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2023/08/22 19:27:00 UTC
[jira] [Commented] (HDDS-3009) Create Container failed using a disk which is full
[ https://issues.apache.org/jira/browse/HDDS-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757635#comment-17757635 ]
Siyao Meng commented on HDDS-3009:
----------------------------------
The question to ask is, what is the expected behavior?
The easiest to do right now might be to just throw "disk is full" error gracefully back to the client. and indicate which DN mount point is full.
Later we could figure out whether the client should ask for a new pipeline. Or, add DN mount point usage metrics to rack awareness so it can hopefully avoid those (almost) full DNs in the first place.
> Create Container failed using a disk which is full
> --------------------------------------------------
>
> Key: HDDS-3009
> URL: https://issues.apache.org/jira/browse/HDDS-3009
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Bharat Viswanadham
> Assignee: Hemant Kumar
> Priority: Major
> Labels: teragentest
>
> /dev/nvme2n1 985G 935G 0 100% /ozone-data
> /dev/nvme4n1 985G 453G 482G 49% /ozone-data1
>
> The container path used is full, but in creating container we have used the disk which is filled up 100% and failing during creating container path creation.
>
> *hdds.datanode.dir is configured with /ozone-data, /ozone*
>
>
> {code:java}
> 2020-02-13 10:58:01,097 ERROR org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil: Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> 2020-02-13 10:58:01,097 INFO org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation: CreateContainer , Trace ID: , Message: Container creation failed. Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container creation failed. Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:177)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:244)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:164)
> at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:412)
> at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:248)
> at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:162)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:396)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:406)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:441)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
> at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:73)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:142)
> ... 12 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org