You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Arafat Khan (Jira)" <ji...@apache.org> on 2026/07/08 12:25:00 UTC
[jira] [Resolved] (HDDS-14577) MISSING_METADATA_DIR trying to mark the container as UNHEALTHY, but creating .container file in metadata dir itself
[ https://issues.apache.org/jira/browse/HDDS-14577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arafat Khan resolved HDDS-14577.
--------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> MISSING_METADATA_DIR trying to mark the container as UNHEALTHY, but creating .container file in metadata dir itself
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-14577
> URL: https://issues.apache.org/jira/browse/HDDS-14577
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 2.0.0
> Reporter: Soumitra Sulav
> Assignee: Priyesh K
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> Error was seen on datanodes as MISSING_METADATA_DIR, but while trying to mark the container as UNHEALTHY, it was creating {{.container}} file in metadata dir itself (which is missing as per the error recorded) and fails with exception ({{{}Failed to create tmp file for /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container{}}})
> {code:java}
> 2026-01-20 09:59:59,929 ERROR [ContainerMetadataScanner]-org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner: Corruption detected in container [69019]. Marking it UNHEALTHY. Container has 1 error: MISSING_METADATA_DIR for file /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata with exception: java.io.FileNotFoundException: Metadata directory /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata not found.
> 2026-01-20 09:59:59,930 WARN [ContainerMetadataScanner]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Unexpected error while marking container 69019 unhealthy
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Failed to create tmp file for /data/2/hadoop-ozone/datanode/data/hdds/CID-40c1b249-f7d7-4287-bbe6-982c3bfb49a8/current/containerDir134/69019/metadata/69019.container: KeyValueContainerData #69019 (UNHEALTHY, non-empty, ri=0, origin=[dn_73ce993f-edab-4819-a73f-b938b0241e48, pipeline_1bb5a281-249c-4262-b299-601d34b4f0bb])
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:318)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:337)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:484)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerState(KeyValueContainer.java:467)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:392)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.markContainerUnhealthy(KeyValueHandler.java:1513)
> at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.markContainerUnhealthy(ContainerController.java:124)
> at org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.handleUnhealthyScanResult(ContainerScanHelper.java:131)
> at org.apache.hadoop.ozone.container.ozoneimpl.ContainerScanHelper.scanMetadata(ContainerScanHelper.java:107)
> at org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner.scanContainer(BackgroundContainerMetadataScanner.java:57)
> at org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.scanContainers(AbstractBackgroundContainerScanner.java:115)
> at org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.runIteration(AbstractBackgroundContainerScanner.java:78)
> at org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.run(AbstractBackgroundContainerScanner.java:59)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: No such file or directory
> at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.base/java.io.File.createTempFile(File.java:2170)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.createTempFile(KeyValueContainer.java:907)
> at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:300)
> ... 13 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org