You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Saketa Chalamchala (Jira)" <ji...@apache.org> on 2025/07/30 19:36:00 UTC
[jira] [Resolved] (HDDS-12422) Make metadata dump path configurable for container export
[ https://issues.apache.org/jira/browse/HDDS-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saketa Chalamchala resolved HDDS-12422.
---------------------------------------
Resolution: Abandoned
> Make metadata dump path configurable for container export
> ---------------------------------------------------------
>
> Key: HDDS-12422
> URL: https://issues.apache.org/jira/browse/HDDS-12422
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Datanode
> Reporter: Saketa Chalamchala
> Assignee: Saketa Chalamchala
> Priority: Major
> Labels: pull-request-available
>
> With SCHEMA_V3, When container is exported the container metadata is dumped from container.db into <containerPath>/metadata/db before getting tar'ed and sent to destination.
> In some edge cases when a volume becomes full (100% utilized), we might want to run balancer but, container replications might fail because there is no more space in the volume to dump metadata files.
> The metadata dump directory needs to be configurable so that metadata can be dumped to a different volume if necessary so that replications from the full volume can succeed.
> {code:java}
> 2025-02-04 18:04:58 INFO KeyValueContainer.packContainerToDestination:1011 - ATTENTION! using v3 path
> 2025-02-04T18:04:58.358333464Z 2025-02-04 18:04:58 INFO KeyValueContainer.packContainerToDestination:1013 - ATTENTION! dumping metadata
> 2025-02-04T18:04:58.365236964Z 2025-02-04 18:04:58 INFO KeyValueContainer.dumpKVContainerDataToFiles:314 - ATTENTION! Creating dir /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db for container 1 to dump metadata to files
> 2025-02-04T18:04:58.365330089Z 2025-02-04 18:04:58 INFO AbstractDatanodeStore.getTableDumpFile:151 - ATTENTION! dumping table 'metadata' under '/data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db'
> 2025-02-04T18:04:58.378313339Z 2025-02-04 18:04:58 WARN PushReplicator.replicate:73 - Container 1 replication was unsuccessful.
> 2025-02-04T18:04:58.378334672Z org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Failed to dump metadata for container 1
> 2025-02-04T18:04:58.378337714Z at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.dumpKVContainerDataToFiles(BlockUtils.java:320)
> 2025-02-04T18:04:58.378342547Z at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.packContainerToDestination(KeyValueContainer.java:1014)
> 2025-02-04T18:04:58.378343714Z at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.exportContainerData(KeyValueContainer.java:719)
> 2025-02-04T18:04:58.378344839Z at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.exportContainer(KeyValueHandler.java:1194)
> 2025-02-04T18:04:58.378346006Z at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.exportContainer(ContainerController.java:195)
> 2025-02-04T18:04:58.378347089Z at org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:62)
> 2025-02-04T18:04:58.378348381Z at org.apache.hadoop.ozone.container.replication.PushReplicator.replicate(PushReplicator.java:67)
> 2025-02-04T18:04:58.378349547Z at org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:82)
> ...
> 2025-02-04T18:04:58.378354922Z at java.base/java.lang.Thread.run(Thread.java:1583)
> 2025-02-04T18:04:58.378356006Z Caused by: java.io.IOException: Failed to finish dumping into file /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data; status : IOError(NoSpace); message : While appending to file: /data/hdds/hdds/CID-232dac26-1ebf-4ca0-bda1-27b75af01e9d/current/containerDir0/1/metadata/db/metadata.data: No space left on device
> ... {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org