You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2021/04/02 04:06:00 UTC
[jira] [Commented] (HDDS-4961) Disk space leak after deleting Ozone
files
[ https://issues.apache.org/jira/browse/HDDS-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313550#comment-17313550 ]
Sammi Chen commented on HDDS-4961:
----------------------------------
Hey [~weichiu], Delete in Ozone is a lazy way. When a key is request to delete, OM will mark the key as deleted and put the blocks related into a pending delete CF. OM will schedule tasks in background, which sends block delete command to SCM. And SCM just do the same as OM. When SCM send delete block command to DN, DN also just record the command and do it lazily. So eventually, the blocks will be deleted, just not that fast.
> Disk space leak after deleting Ozone files
> ------------------------------------------
>
> Key: HDDS-4961
> URL: https://issues.apache.org/jira/browse/HDDS-4961
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> I'm finding cases where deleting Ozone files does not reclaim disk space, even if the corresponding containers are empty and in CLOSED state.
> In some cases, blocks are removed from container, resulting in empty and closed container. However, the container metadata is not removed, and SCM still records the container. No way to remove them. In this case, each dangling container occupies ~140mb of space. Overtime, tens or even hundreds of GB is wasted on each DN.
> Because of HDDS-2534, there is no way to manually delete the containers.
> Steps to reproduce:
> {code}
> ozone sh volume create o3://ozone1/vol1
> ozone sh bucket create o3://ozone1/vol1/bucket1
> # Generated data.
> export EXAMPLES_PATH=
> yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce//hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096 -Dmapreduce.reduce.memory.mb=4096 1000000000 o3fs://bucket1.vol1.ozone1/teragen11
> Delete skip trash:
> hdfs dfs -rm -r -skipTrash "o3fs://bucket1.vol1.ozone1/*”
> (run "ozone admin container close ${containerid} to force closing the containers)
> # check disk usage:
> du -h /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> # the container directory is not removed
> ...
> 142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/34-dn-container.db
> 4.0K /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/db.checkpoints
> 142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata
> 4.0K /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/chunks
> 142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34
> 13G /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org